Data processing method and device, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, a readable storage medium and a program product. The method comprises the steps that under the condition that a second node has a fault, multiple pieces of first data to be cached and reconstructed included in a first log in a first node are determined, the first data are written into a ferry pool in the first node, and then prompt information can be output, the multiple pieces of first data are the same as multiple pieces of second data included in a second log in the second node, and the prompt information is used for prompting that cache reconstruction is completed. By adopting the method, the service recovery efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method, device, readable storage medium and program product. Background Art

[0002] In the storage system, small block write refers to the operation of writing relatively small data blocks to storage media (such as hard disk drives, solid-state drives (SSDs), etc.). Usually, the host node sends a small block write instruction to the current device node where the storage system is deployed. The current device node writes the corresponding data into the log, parses the data in the log to generate the corresponding cache instance, and then writes the cache instance into the cache, which can respond to the host node's write completion. Next, the cache instance in the cache is solidified to the storage medium to complete the data storage.

[0003] When the current device node fails, the service is interrupted. At this time, the backup device node is usually enabled, and the backup device node replaces the current device node to create a new cache instance, and then writes the cache instance into the cache to realize the cache reconstruction process.

[0004] However, the above method has the problem of low cache reconstruction efficiency, which in turn affects business recovery efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, device, readable storage medium and program product that can improve business recovery efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] In the case that the second node fails, determining a plurality of first data to be cached and reconstructed included in the first log in the first node, the plurality of first data being the same as a plurality of second data included in the second log in the second node;

[0008] Writing each first data into the ferry pool in the first node;

[0009] Output prompt information, which is used to indicate that cache rebuilding is complete.

[0010] In the above embodiment, when the second node fails, the first node serving as a backup can directly obtain the first data to be cached and rebuilt that is the same as that in the second log of the second node from the first log. After the first node writes these first data to the ferry pool, it can output a prompt that the cache reconstruction is completed. The host-side node can immediately start issuing small block write instructions, thereby shortening the cache reconstruction completion cycle and improving business recovery efficiency.

[0011] In one embodiment, writing each first data into a ferry pool in the memory of the first node includes:

[0012] For each first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data;

[0013] The first data is written into the ferry pool according to the first mapping address.

[0014] In the above embodiment, each first data is mapped to obtain a first mapping address corresponding to each first data, and each first data is written into the ferry pool according to the first mapping address corresponding to each first data, so that each first data can be quickly found by using the first mapping address of each first data, which facilitates the management of the first data.

[0015] In one embodiment, the ferry pool includes multiple memory spaces, and writing the first data into the ferry pool according to the first mapping address includes:

[0016] Matching the first mapping address with the mapping addresses corresponding to each memory space;

[0017] The first data is written into the memory space corresponding to the successfully matched mapping address.

[0018] In the above embodiment, the ferry pool is divided into multiple memory spaces with different mapping addresses, and the first node writes the first data with the same first mapping address into the same memory space, which facilitates subsequent data reading and improves the efficiency of cache reconstruction.

[0019] In one embodiment, the method further comprises:

[0020] Receive a write instruction sent by a host node, the write instruction carrying initial write data;

[0021] When the transition memory cleanup time has not arrived, detect whether the ferry pool is empty and obtain the detection result;

[0022] According to the detection result, the cache instance corresponding to the initial write data is written into the cache of the first node.

[0023] In the above embodiment, the first node processes the first data in the ferry pool while executing the write instruction sent by the host node, and performs cache reconstruction without perception, thereby improving the cache reconstruction efficiency and enhancing the user experience.

[0024] In one embodiment, writing the cache instance corresponding to the initial write data into the cache of the first node according to the detection result includes:

[0025] If the detection result shows that the ferry pool is not empty, the initial written data is mapped according to a preset mapping method to obtain a second mapping address corresponding to the initial written data;

[0026] Determine the target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and match the initial write data with the target first data in the target memory space if the target memory space is not empty;

[0027] If the initial write data matches the target first data, the target first data is deleted in the target memory space, and the cache instance corresponding to the initial write data is written into the cache of the first node.

[0028] In the above embodiment, when the ferry pool is not empty, the initial write data is first mapped, so as to find the target memory space according to the second mapping address obtained by the mapping process, and then detect whether the memory space is empty. If it is not empty, the initial write data is matched with the target first data, and the cache instance corresponding to the initial write data is written into the cache according to the matching result, thereby avoiding unnecessary computing resource overhead.

[0029] In one embodiment, the matching of the initial write data and the target first data indicates that the initial write data and the target first data are completely identical, and writing the cache instance corresponding to the initial write data into the cache of the first node includes:

[0030] Generate a cache instance based on the initial write data;

[0031] Write the cache instance to the cache of the first node.

[0032] In the above embodiment, for completely identical first data, the first node can directly delete the target first data in the ferry pool, and write the cache instance corresponding to the initial write data into the cache of the first node, thereby completing the cache reconstruction of the target first data while executing the write instruction on the initial write data, thereby reducing the overhead of computing resources.

[0033] In one embodiment, the matching of the initial write data and the target first data indicates that the initial write data and the target first data are partially identical, and writing the cache instance corresponding to the initial write data into the cache of the first node includes:

[0034] According to the part of the target first data that is different from the initial written data, the initial written data is supplemented to obtain the target written data;

[0035] A cache instance is generated according to the target write data, and the cache instance is written into the cache of the first node.

[0036] In the above embodiment, for partially identical target first data, the first node first uses the target first data to fill in the gaps in the initial write data, and the obtained target write data includes the complete target first data and the initial write data. The cache instance corresponding to the target write data is written into the cache corresponding to the first node, thereby ensuring the integrity of the data.

[0037] In one embodiment, the method further comprises:

[0038] If the initial write data does not match the target first data, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0039] In the above embodiment, the first node does not need to process the unmatched target first data, and can directly write the cache instance corresponding to the initial write data into the cache, thereby improving the efficiency of data writing.

[0040] In one embodiment, according to the detection result, writing the cache instance corresponding to the initial write data into the cache of the first node also includes:

[0041] If the detection result is that the ferry pool is empty, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0042] In the above embodiment, it is first detected whether the ferry pool is empty, and then subsequent slave data processing is performed, thereby avoiding unnecessary resource waste caused by the first node directly comparing each first data according to each initially written data.

[0043] In one embodiment, the method further comprises:

[0044] If the transition memory cleanup time is reached, the cache instances corresponding to the first data in the ferry pool are written into the cache of the first node, and the first data are deleted in the ferry pool.

[0045] In the above embodiment, after the preset transition memory cleanup time is reached, the cache of the first data in the ferry pool will be directly rebuilt to avoid the first data being omitted and not being rebuilt for a long time, thereby improving the security of the data.

[0046] In a second aspect, the present application further provides a data processing device, comprising:

[0047] a determination module, configured to determine, when a fault occurs in the second node, that a plurality of first data to be cached and reconstructed included in a first log in the first node are the same as a plurality of second data included in a second log in the second node;

[0048] A transition module, used for writing each first data into a ferry pool in the first node;

[0049] The prompt module is used to output prompt information, and the prompt information is used to prompt that the cache reconstruction is completed.

[0050] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect described above.

[0052] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in the first aspect are implemented.

[0053] The above-mentioned data processing method, device, readable storage medium and program product, when there is a fault in the second node, determine the multiple first data to be cached and rebuilt included in the first log of the first node, and write each first data into the ferry pool in the first node, and output a prompt message, wherein the multiple first data are the same as the multiple second data included in the second log in the second node, and the prompt message is used to prompt that the cache reconstruction is completed. In this way, when there is a fault in the second node, the first node as a backup can directly obtain the first data to be cached and rebuilt that are the same as those in the second log of the second node from the first log. After the first node writes these first data to the ferry pool, it can output a prompt that the cache reconstruction is complete, and the host-end node can immediately start issuing small block write instructions, which shortens the cache reconstruction completion cycle and improves business recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 Structural diagrams written for small blocks in traditional technology;

[0056] Figure 2 An application environment diagram of a data processing method in an embodiment;

[0057] Figure 3 is a flow chart of a data processing method in one embodiment;

[0058] Figure 4 is a flow chart of step 302 in another embodiment;

[0059] Figure 5 A schematic diagram of a process of writing first data into a ferry pool in another embodiment;

[0060] Figure 6 It is a schematic diagram of a flow chart of processing a new write instruction sent to a host node in another embodiment;

[0061] Figure 7 A schematic diagram of a process of rebuilding the first data cache in the ferry pool when a preset transition memory cleanup time is reached in another embodiment;

[0062] Figure 8 A schematic diagram of a flow chart of step 603 in another embodiment;

[0063] Fig. 9 A schematic diagram of a flow chart of executing a write instruction when the target memory space is empty in another embodiment;

[0064] Fig.10 It is a flowchart of executing a write instruction when the target memory space is not empty in another embodiment;

[0065] Fig.11 is a flowchart of an exemplary data processing method in another embodiment;

[0066] Fig.12 is a structural block diagram of a data processing device in one embodiment;

[0067] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] In a storage system, small block write refers to the operation of writing relatively small data blocks to a storage medium (such as a hard disk drive, solid-state drive SSD, etc.).

[0070] The process of writing small blocks can be referred to Figure 1Usually, the host node sends a small block write instruction to the current device node where the storage system is deployed. The current device node writes the corresponding data into the log, parses the data in the log to generate the corresponding cache instance, and then writes the cache instance into the cache, which can respond to the completion of the host node writing. Next, the cache instance in the cache is solidified into the storage medium to complete the storage of the data.

[0071] When the current device node fails, the business is interrupted. At this time, the backup device node is usually enabled. The log in the backup device node is synchronized with the log in the current device node. The backup device node parses the data in the log to create a new cache instance, and then writes the cache instance to the cache to complete the cache reconstruction. After the entire cache reconstruction process is completed, the backup device node will output information indicating that the cache reconstruction is complete.

[0072] During the cache reconstruction process, if the amount of data in the log that needs to be rebuilt is too large, the standby device node will need to spend a long time parsing the data in the log and creating a new cache instance, resulting in the entire cache reconstruction cycle being too long. During the cache reconstruction process, the host node cannot perform new small block write operations, resulting in low business recovery efficiency.

[0073] In view of this, the embodiments of the present application provide a data processing method, device, readable storage medium and program product. When a fault occurs in the second node, multiple first data to be cached and rebuilt included in the first log of the first node are determined, and each first data is written into the ferry pool in the first node, and a prompt message can be output, wherein the multiple first data are the same as the multiple second data included in the second log in the second node, and the prompt message is used to prompt that the cache reconstruction is complete. In this way, when a fault occurs in the second node, the first node as a backup can directly obtain the first data to be cached and rebuilt that are the same as those in the second log of the second node from the first log. After the first node writes these first data into the ferry pool, it can output a prompt that the cache reconstruction is complete, and the host-end node can immediately start issuing small block write instructions, which shortens the cache reconstruction completion cycle and improves business recovery efficiency.

[0074] The data processing method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown, the host node 200 can communicate with the first node 201, and the host node can also communicate with the second node 202. The storage system deployed on the first node 201 and the second node 202 is the same. When the second node 202 operates normally, the second log in the second node 202 will synchronize data with the first log in the first node 201.

[0075] In an exemplary embodiment, Figure 3 As shown, a data processing method is provided, which is applied to Figure 2 Taking the first node 201 in the example as an example, the method includes the following steps 301 to 303. Among them:

[0076] Step 301 : when a second node fails, determine a plurality of first data to be cached and reconstructed included in a first log in a first node.

[0077] The multiple first data are the same as the multiple second data included in the second log in the second node.

[0078] In an embodiment of the present application, when the second node operates normally, when the second node stores the data sent by the host node in the storage medium, it synchronizes the second data in the second log with the first log in the first node, so that the first data in the first log is the same as the second data in the second log.

[0079] When the second node fails, the first node will be enabled to continue providing storage services to the host node.

[0080] In a possible implementation, when the second node fails, the first node receives a startup instruction sent by the host node, and the first node can provide storage services for the host node according to the startup instruction.

[0081] In another possible implementation, when the second node fails, the first node will automatically start, so that the host-side node continues to provide storage services.

[0082] It is understandable that the timing of the failure of the second node is uncontrollable. When the second node fails, its cache may still store data that has not yet been solidified to the storage medium. This data is the same as the second data in the second log. Therefore, when the first node is performing a cache reconstruction process, it is necessary to determine the first data included in the first log that is the same as the second data in the second log, and use the first data to perform cache reconstruction.

[0083] Step 302: write each first data into the ferry pool in the first node.

[0084] In order to shorten the cache reconstruction cycle, after the first node determines a plurality of first data that need to be cached and reconstructed, the first data may be written into the ferry pool first.

[0085] In an embodiment of the present application, a ferry pool is deployed in the first node, and the ferry pool includes a preset storage space. The storage space is different from the cache in the first node, and can be a storage space in the memory or other storage space. The first node does not need to parse the first data and can directly write the first data into the ferry pool. Therefore, using the ferry pool to transition the first data to the cache can reduce the time for cache reconstruction.

[0086] In a possible implementation, in order to facilitate subsequent data processing, the storage space included in the ferry pool can be divided into multiple subspaces, and the first node can store each first data in different subspaces according to different classification methods. Optionally, the first node can obtain the hot and cold levels corresponding to each first data, and then write the first data with similar hot and cold levels into the same subspace; optionally, the first node can hash each first data according to a preset hash condition to obtain a hash value corresponding to each first data, and then write the first data with the same hash value into the same subspace.

[0087] Step 303: output prompt information.

[0088] The prompt information is used to prompt that the cache rebuild is completed.

[0089] In an embodiment of the present application, after the first node writes each first data into the ferry pool, it can directly output a prompt message to prompt that the cache reconstruction is complete, and the host-end node can perform subsequent storage operations normally, shortening the cache reconstruction cycle and improving the efficiency of business recovery. For the first data directly written into the ferry pool, in a possible implementation mode, the first node can parse each first data in a preset manner and create a corresponding cache instance while the subsequent host-end node performs normal storage operations, and write the cache instance into the cache of the first node, and then clear the corresponding first data in the ferry pool. By repeating the above process, the first node can empty the first data in the ferry pool and complete the cache reconstruction.

[0090] In another possible implementation, the first node may wait for a suitable cache reconstruction time according to preset conditions. Optionally, the first node does not receive a storage operation issued by the host node within a preset time length. For example, the first node does not receive a storage operation issued by the host node within 30 minutes. At this time, the first node may parse each first data in the ferry pool, create a corresponding cache instance, write the cache instance into the cache of the first node, and delete each first data in the ferry pool.

[0091] In an embodiment of the present application, in order to avoid the ferry pool being occupied for a long time, that is, the first data in the ferry pool has not been cached and rebuilt for a long time, the first node can set a transition memory cleanup time. If the transition memory cleanup time is reached, the first node will write the cache instance corresponding to each first data in the ferry pool into the cache of the first node, and delete each first data in the ferry pool, thereby avoiding the existence of omitted first data that has not been cached and rebuilt for a long time, thereby improving data security.

[0092] In the above embodiment, when the second node fails, the first node serving as a backup can directly obtain the first data to be cached and rebuilt that is the same as that in the second log of the second node from the first log. After the first node writes these first data to the ferry pool, it can output a prompt that the cache reconstruction is completed. The host-side node can immediately start issuing small block write instructions, thereby shortening the cache reconstruction completion cycle and improving business recovery efficiency.

[0093] In one embodiment, based on the above Figure 3 The embodiment shown, see Figure 4 , this embodiment involves a process of writing each first data into a ferry pool in the memory of the first node. Figure 4 As shown, step 302 may include step 401 and step 402 .

[0094] Step 401 : for each first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data.

[0095] In the embodiment of the present application, in order to facilitate the subsequent reading of each first data from the ferry pool, the first node can store each first data in the ferry pool according to different mapping addresses.

[0096] Regarding the method of obtaining the mapping address, optionally, the preset mapping method can be a preset encoding method, such as a preset hash formula, and the first node can perform encoding conversion processing on each first data according to the preset encoding method to obtain the encoding corresponding to each first data, and the encoding is the first mapping address corresponding to the first data; optionally, the preset mapping method can be a trained mapping model, and the first node can input each first data into the mapping model to obtain the first mapping address corresponding to each first data output by the mapping model.

[0097] Step 402: write the first data into the ferry pool according to the first mapping address.

[0098] After obtaining the first mapping addresses corresponding to the first data, the first node may write the first data into the ferry pool according to the first mapping addresses.

[0099] In a possible implementation, the ferry pool includes multiple memory spaces, wherein different memory spaces may correspond to different mapping addresses, so that the process of the first node writing the first data into the ferry pool may include: the first node matches the first mapping address with the mapping addresses corresponding to each memory space, and then the first node writes the first data into the memory space corresponding to the successfully matched mapping address.

[0100] In this way, the ferry pool is divided into multiple memory spaces with different mapping addresses, and the first node writes the first data with the same first mapping address into the same memory space, which facilitates subsequent data reading and improves the efficiency of cache reconstruction.

[0101] For example, refer to Figure 5 The ferry pool may include multiple memory spaces, and each memory space corresponds to a different mapping address. After the first node determines the memory space corresponding to each first data, it may write each first data into each memory space in sequence.

[0102] In the above embodiment, each first data is mapped to obtain a first mapping address corresponding to each first data, and each first data is written into the ferry pool according to the first mapping address corresponding to each first data, so that each first data can be quickly found by using the first mapping address of each first data, which facilitates the management of the first data.

[0103] In one embodiment, based on the above Figure 3 The embodiment shown, see Figure 6 This embodiment involves the process of processing a new write instruction sent by a host node. Figure 6 As shown, the process may include steps 601 to 603.

[0104] Step 601, receiving a write instruction sent by a host node.

[0105] In an embodiment of the present application, after the first node outputs the prompt information, it may receive a write instruction sent by the stage node, wherein the write instruction carries initial write data, which may be brand new data or data used to update old data, that is, the initial write data may be repeated with or different from each first data. Based on this, the first node can rebuild the cache of the first data stored in the ferry pool while executing the write instruction sent by the host node.

[0106] Step 602, when the transition memory cleanup time has not been reached, detect whether the ferry pool is empty and obtain the detection result.

[0107] In a possible implementation, after receiving the write instruction sent by the host node, the first node may first detect whether the current time reaches the preset transition memory cleanup time. Optionally, if the preset transition memory cleanup time is reached, it means that the time for the current first data to be written into the ferry pool is too long, and the first data has not been cached and rebuilt for a long time. Then, the first node may directly perform cache reconstruction on each first data. For example, referring to Figure 7 The first node reads out each first data, parses each first data and creates a corresponding cache instance, writes the cache instance into the cache of the first node, and clears the first data in the ferry pool, thereby completing the cache reconstruction of all first data.

[0108] Optionally, if the transition memory cleanup time is not reached, it means that the cache reconstruction of each first data is not urgent. The first node can first detect whether the ferry pool is empty, and execute the write instruction according to the detection result, that is, write the cache instance corresponding to the initial write data into the cache of the first node.

[0109] Step 603: Write the cache instance corresponding to the initial write data into the cache of the first node according to the detection result.

[0110] Optionally, if the detection result is that the ferry pool is empty, it means that the first data in the ferry pool has been cached and rebuilt, and the first node does not need to process the data in the ferry pool. That is, if the detection result is that the ferry pool is empty, the first node generates a cache instance based on the initial write data and writes the cache instance into the cache of the first node.

[0111] In this way, it is first detected whether the ferry pool is empty, and then subsequent slave data processing is performed, thereby avoiding unnecessary resource waste caused by the first node directly comparing each first data according to each initially written data.

[0112] Optionally, if the detection result is that the ferry pool is not empty, it means that there is still first data in the ferry pool that has not been cached and rebuilt. It can be understood that the first data may or may not have overlapping content with the initially written data. In one possible implementation, the first node may compare the initially written data with each first data to determine whether there is first data that overlaps with the initially written data. If so, the first node may delete the overlapping data in the ferry pool and write the cache instance corresponding to the initially written data into the cache of the first node.

[0113] Based on the above embodiments, in a possible implementation, when all the first data in the ferry pool are cached and rebuilt, the first node can generate a reconstruction completion instruction, which is used to indicate that no operations will be performed on the ferry pool, thereby avoiding the situation where the first data has been cached and rebuilt, and the first node still continuously checks whether the ferry pool is empty, thereby reducing unnecessary waste of resources.

[0114] In the above embodiment, the first node processes the first data in the ferry pool while executing the write instruction sent by the host node, and performs cache reconstruction without perception, thereby improving the cache reconstruction efficiency and enhancing the user experience.

[0115] In one embodiment, based on the above Figure 6 The embodiment shown, see Figure 8 This embodiment relates to a process of writing the cache instance corresponding to the initial write data into the cache of the first node according to the detection result. Figure 8 As shown, step 603 may include steps 801 to 803 .

[0116] Step 801: If the detection result is that the ferry pool is not empty, the initial written data is mapped according to a preset mapping method to obtain a second mapping address corresponding to the initial written data.

[0117] In an embodiment of the present application, if the detection result is that the ferry pool is not empty, it means that there is still first data in the ferry pool that has not been cached and rebuilt. It can be understood that the first data may or may not overlap with the initially written data. The first node can then detect the remaining first data in the ferry pool based on the initially written data to determine whether there is first data that overlaps with the initially written data.

[0118] In an embodiment of the present application, the first node can map the initial write data according to a preset mapping method to obtain a second mapping address corresponding to the initial write data, wherein the preset mapping method is the same as the preset mapping method for the first data mentioned above.

[0119] Regarding the mapping process, reference may be made to the process of the first node mapping each first data in the above embodiment, which will not be described in detail here.

[0120] Step 802, determining the target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and matching the initial write data with the target first data in the target memory space when the target memory space is not empty.

[0121] After the first node obtains the second mapping address corresponding to the initial write data, it can determine the target memory space corresponding to the initial write data in the ferry pool based on the second mapping address. It can be understood that due to the same preset mapping method, the first data stored in the target memory space may have content overlap with the initial write data, while for the first data stored in other memory spaces, the mapping address values ​​obtained by mapping are different, and the content cannot overlap. The first node does not need to consider matching the initial write data with the first data in other memory spaces.

[0122] In an embodiment of the present application, in order to avoid unnecessary waste of resources, the first node may first detect whether the target memory space is empty.

[0123] In a possible implementation manner, if the target memory space is empty, it means that there is no first data in the target storage space that may overlap with the initially written data content. Fig. 9 , the first node can directly write the cache instance corresponding to the initial write data into the cache of the first node.

[0124] In another possible implementation, if the target memory space is not empty, it means that there may be first data in the target storage space that overlaps with the initially written data content, and the first node may match the initially written data with the target first data in the target memory space.

[0125] It can be understood that the number of target first data can be multiple.

[0126] Optionally, if the initial write data does not match the target first data, indicating that the initial write data is completely different from the target first data, the first node may generate a cache instance according to the initial write data and write the cache instance into the cache of the first node.

[0127] In this way, the first node does not need to process the unmatched target first data, and can directly write the cache instance corresponding to the initial write data into the cache, thereby improving the efficiency of data writing.

[0128] Step 803: If the initial write data matches the target first data, the target first data is deleted in the target memory space, and the cache instance corresponding to the initial write data is written into the cache of the first node.

[0129] In the embodiment of the present application, if the initial write data matches the target first data, it means that there is content overlap between the initial write data and the target first data.

[0130] Reference Fig.10 In a possible implementation, the initial write data matches the target first data, indicating that the initial write data is exactly the same as the target first data. The first node can directly generate a cache instance based on the initial write data and write the cache instance into the cache of the first node.

[0131] In this way, for completely identical target first data, the first node can directly delete the target first data in the ferry pool, and write the cache instance corresponding to the initial write data into the cache of the first node, and complete the cache reconstruction of the target first data while executing the write instruction on the initial write data, thereby reducing the overhead of computing resources.

[0132] In another possible implementation, the initial write data matches the target first data, indicating that the initial write data is partially identical to the target first data. The first node may fill in the gaps in the initial write data based on the portion of the target first data that is different from the initial write data to obtain the target write data. Then, a cache instance is generated based on the target write data, and the cache instance is written into the cache of the first node.

[0133] In this way, for partially identical target first data, the first node first uses the target first data to fill in the gaps in the initial write data. The resulting target write data includes the complete target first data and the initial write data. The cache instance corresponding to the target write data is written into the cache corresponding to the first node, ensuring the integrity of the data.

[0134] In the above embodiment, when the ferry pool is not empty, the initial write data is first mapped, so as to find the target memory space according to the second mapping address obtained by the mapping process, and then detect whether the memory space is empty. If it is not empty, the initial write data is matched with the target first data, and the cache instance corresponding to the initial write data is written into the cache according to the matching result, thereby avoiding unnecessary computing resource overhead.

[0135] In one embodiment, referring to Fig.11 , provides an exemplary data processing method, which can be applied to Figure 1 The first node in .

[0136] Step 1101: when a second node fails, determine a plurality of first data to be cached and reconstructed included in a first log in a first node.

[0137] The multiple first data are the same as the multiple second data included in the second log in the second node.

[0138] Step 1102 : For each first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data.

[0139] Among them, the ferry pool includes multiple memory spaces.

[0140] Step 1103, matching the first mapping address with the mapping addresses corresponding to each memory space.

[0141] Step 1104, writing the first data into the memory space corresponding to the successfully matched mapping address.

[0142] Step 1105, output prompt information.

[0143] The prompt information is used to prompt that the cache rebuild is completed.

[0144] Step 1106, receiving a write instruction sent by the host node.

[0145] The write instruction carries the initial write data.

[0146] Step 1107: if the transition memory cleanup time is reached, the cache instances corresponding to the first data in the ferry pool are written into the cache of the first node, and the first data in the ferry pool are deleted.

[0147] Step 1108, when the transition memory cleanup time has not been reached, detect whether the ferry pool is empty and obtain the detection result.

[0148] Step 1109: If the detection result is that the ferry pool is empty, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0149] Step 1110, if the detection result is that the ferry pool is not empty, the initial write data is mapped according to a preset mapping method to obtain a second mapping address corresponding to the initial write data.

[0150] Step 1111, determine the target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and when the target memory space is not empty, match the initial write data with the target first data in the target memory space.

[0151] Step 1112: If the initial write data does not match the target first data, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0152] Step 1113 : if the initial write data matches the target first data, indicating that the initial write data and the target first data are completely identical, a cache instance is generated according to the initial write data.

[0153] Step 1114: write the cache instance into the cache of the first node.

[0154] Step 1115, if the initial write data matches the target first data, indicating that the initial write data and the target first data are partially identical, then the initial write data is supplemented according to the part of the target first data that is different from the initial write data to obtain the target write data;

[0155] Step 1116: Generate a cache instance according to the target write data, and write the cache instance into the cache of the first node.

[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0158] In an exemplary embodiment, Fig.12 As shown, a data processing device is provided, which is used for a first node, including: a determination module 1201, a transition module 1202 and a prompt module 1203, wherein:

[0159] The determination module 1201 is used to determine, when a fault occurs in the second node, that a plurality of first data to be cached and reconstructed included in a first log in the first node are the same as a plurality of second data included in a second log in the second node;

[0160] A transition module 1202, configured to write each first data into a ferry pool in the first node;

[0161] The prompt module 1203 is used to output prompt information, where the prompt information is used to prompt that the cache reconstruction is completed.

[0162] In one embodiment, the transition module 1202 includes:

[0163] A first mapping unit is used to map each first data according to a preset mapping method to obtain a first mapping address corresponding to the first data;

[0164] The transition writing unit is used to write the first data into the ferry pool according to the first mapping address.

[0165] In one embodiment, the ferry pool includes multiple memory spaces, and the transition writing unit is specifically used to execute:

[0166] Matching the first mapping address with the mapping addresses corresponding to each memory space;

[0167] The first data is written into the memory space corresponding to the successfully matched mapping address.

[0168] In one embodiment, the data processing device further comprises:

[0169] The instruction receiving module is used to receive the write instruction sent by the host node, and the write instruction carries the initial write data;

[0170] The overall memory detection module is used to detect whether the ferry pool is empty and obtain the detection result when the transition memory cleanup time has not been reached;

[0171] The writing module is used to write the cache instance corresponding to the initial write data into the cache of the first node according to the detection result.

[0172] In one embodiment, the write module includes:

[0173] The second processing unit is used for mapping the initial written data according to a preset mapping method to obtain a second mapping address corresponding to the initial written data if the detection result shows that the ferry pool is not empty;

[0174] A matching unit, used to determine the target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and match the initial write data with the target first data in the target memory space when the target memory space is not empty;

[0175] The reconstruction write unit is used to delete the target first data in the target memory space if the initial write data matches the target first data, and write the cache instance corresponding to the initial write data into the cache of the first node.

[0176] In one embodiment, the matching of the initial written data and the target first data indicates that the initial written data and the target first data are completely identical, and the reconstruction writing unit is specifically used to perform:

[0177] Generate a cache instance based on the initial write data;

[0178] Write the cache instance to the cache of the first node.

[0179] In one embodiment, the matching of the initial write data and the target first data indicates that the initial write data and the target first data are partially identical, and the reconstruction write unit is further configured to execute:

[0180] According to the part of the target first data that is different from the initial written data, the initial written data is supplemented to obtain the target written data;

[0181] A cache instance is generated according to the target write data, and the cache instance is written into the cache of the first node.

[0182] In one embodiment, the writing module further includes:

[0183] The first direct write unit is used to generate a cache instance according to the initial write data and write the cache instance into the cache of the first node if the initial write data does not match the target first data.

[0184] In one embodiment, the writing module further includes:

[0185] The second direct write unit is used to generate a cache instance according to the initial write data and write the cache instance into the cache of the first node if the detection result is that the ferry pool is empty.

[0186] In one embodiment, the data processing device further comprises:

[0187] The direct reconstruction module is used to write the cache instance corresponding to each first data in the ferry pool into the cache of the first node if the transition memory cleaning time is reached, and delete each first data in the ferry pool.

[0188] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0189] In an exemplary embodiment, a computer device is provided. The computer device may be a first node, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0190] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0191] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0192] In the case that the second node fails, determining a plurality of first data to be cached and reconstructed included in a first log in the first node, where the plurality of first data are the same as a plurality of second data included in a second log in the second node;

[0193] Writing each of the first data into the ferry pool in the first node;

[0194] Output prompt information, where the prompt information is used to prompt that the cache rebuild is completed.

[0195] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0196] For each of the first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data;

[0197] The first data is written into the ferry pool according to the first mapping address.

[0198] In one embodiment, the ferry pool includes multiple memory spaces, and the processor further implements the following steps when executing the computer program:

[0199] Matching the first mapping address with the mapping addresses corresponding to the memory spaces;

[0200] The first data is written into the memory space corresponding to the successfully matched mapping address.

[0201] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0202] Receiving a write instruction sent by a host node, wherein the write instruction carries initial write data;

[0203] When the transition memory cleanup time has not been reached, detecting whether the ferry pool is empty and obtaining a detection result;

[0204] According to the detection result, the cache instance corresponding to the initial write data is written into the cache of the first node.

[0205] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0206] If the detection result is that the ferry pool is not empty, mapping the initial written data according to a preset mapping method to obtain a second mapping address corresponding to the initial written data;

[0207] Determine a target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and match the initial write data with the target first data in the target memory space if the target memory space is not empty;

[0208] If the initial write data matches the target first data, the target first data is deleted in the target memory space, and the cache instance corresponding to the initial write data is written into the cache of the first node.

[0209] In one embodiment, the matching between the initial written data and the target first data indicates that the initial written data and the target first data are completely identical, and when the processor executes the computer program, the following steps are further implemented:

[0210] Generate the cache instance according to the initial write data;

[0211] The cache instance is written into the cache of the first node.

[0212] In one embodiment, the matching between the initial written data and the target first data indicates that the initial written data and the target first data are partially identical, and when the processor executes the computer program, the following steps are further implemented:

[0213] According to the part of the target first data that is different from the initial written data, the initial written data is supplemented to obtain the target written data;

[0214] The cache instance is generated according to the target write data, and the cache instance is written into the cache of the first node.

[0215] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0216] If the initial write data does not match the target first data, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0217] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0218] If the detection result is that the ferry pool is empty, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0219] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0220] If the transition memory cleanup time is reached, the cache instances corresponding to the first data in the ferry pool are written into the cache of the first node, and the first data in the ferry pool are deleted.

[0221] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0222] In the case that the second node fails, determining a plurality of first data to be cached and reconstructed included in a first log in the first node, where the plurality of first data are the same as a plurality of second data included in a second log in the second node;

[0223] Writing each of the first data into the ferry pool in the first node;

[0224] Output prompt information, where the prompt information is used to prompt that the cache rebuild is completed.

[0225] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0226] For each of the first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data;

[0227] The first data is written into the ferry pool according to the first mapping address.

[0228] In one embodiment, the ferry pool includes multiple memory spaces, and when the computer program is executed by the processor, the following steps are also implemented:

[0229] Matching the first mapping address with the mapping addresses corresponding to the memory spaces;

[0230] The first data is written into the memory space corresponding to the successfully matched mapping address.

[0231] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0232] Receiving a write instruction sent by a host node, wherein the write instruction carries initial write data;

[0233] When the transition memory cleanup time has not been reached, detecting whether the ferry pool is empty and obtaining a detection result;

[0234] According to the detection result, the cache instance corresponding to the initial write data is written into the cache of the first node.

[0235] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0236] If the detection result is that the ferry pool is not empty, mapping the initial written data according to a preset mapping method to obtain a second mapping address corresponding to the initial written data;

[0237] Determine a target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and match the initial write data with the target first data in the target memory space if the target memory space is not empty;

[0238] If the initial write data matches the target first data, the target first data is deleted in the target memory space, and the cache instance corresponding to the initial write data is written into the cache of the first node.

[0239] In one embodiment, the matching between the initial written data and the target first data indicates that the initial written data and the target first data are completely identical, and when the computer program is executed by the processor, the following steps are further implemented:

[0240] Generate the cache instance according to the initial write data;

[0241] The cache instance is written into the cache of the first node.

[0242] In one embodiment, the matching between the initial written data and the target first data indicates that the initial written data and the target first data are partially identical, and when the computer program is executed by the processor, the following steps are further implemented:

[0243] According to the part of the target first data that is different from the initial written data, the initial written data is supplemented to obtain the target written data;

[0244] The cache instance is generated according to the target write data, and the cache instance is written into the cache of the first node.

[0245] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0246] If the initial write data does not match the target first data, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0247] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0248] If the detection result is that the ferry pool is empty, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0249] If the transition memory cleanup time is reached, the cache instances corresponding to the first data in the ferry pool are written into the cache of the first node, and the first data in the ferry pool are deleted.

[0250] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0251] In the case that the second node fails, determining a plurality of first data to be cached and reconstructed included in a first log in the first node, where the plurality of first data are the same as a plurality of second data included in a second log in the second node;

[0252] Writing each of the first data into the ferry pool in the first node;

[0253] Output prompt information, where the prompt information is used to prompt that the cache rebuild is completed.

[0254] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0255] For each of the first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data;

[0256] The first data is written into the ferry pool according to the first mapping address.

[0257] In one embodiment, the ferry pool includes multiple memory spaces, and when the computer program is executed by the processor, the following steps are also implemented:

[0258] Matching the first mapping address with the mapping addresses corresponding to the memory spaces;

[0259] The first data is written into the memory space corresponding to the successfully matched mapping address.

[0260] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0261] Receiving a write instruction sent by a host node, wherein the write instruction carries initial write data;

[0262] When the transition memory cleanup time has not been reached, detecting whether the ferry pool is empty and obtaining a detection result;

[0263] According to the detection result, the cache instance corresponding to the initial write data is written into the cache of the first node.

[0264] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0265] If the detection result is that the ferry pool is not empty, mapping the initial written data according to a preset mapping method to obtain a second mapping address corresponding to the initial written data;

[0266] Determine a target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and match the initial write data with the target first data in the target memory space if the target memory space is not empty;

[0267] If the initial write data matches the target first data, the target first data is deleted in the target memory space, and the cache instance corresponding to the initial write data is written into the cache of the first node.

[0268] In one embodiment, the matching between the initial written data and the target first data indicates that the initial written data and the target first data are completely identical, and when the computer program is executed by the processor, the following steps are further implemented:

[0269] Generate the cache instance according to the initial write data;

[0270] The cache instance is written into the cache of the first node.

[0271] In one embodiment, the matching between the initial written data and the target first data indicates that the initial written data and the target first data are partially identical, and when the computer program is executed by the processor, the following steps are further implemented:

[0272] According to the part of the target first data that is different from the initial written data, the initial written data is supplemented to obtain the target written data;

[0273] The cache instance is generated according to the target write data, and the cache instance is written into the cache of the first node.

[0274] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0275] If the initial write data does not match the target first data, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0276] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0277] If the detection result is that the ferry pool is empty, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

[0278] If the transition memory cleanup time is reached, the cache instances corresponding to the first data in the ferry pool are written into the cache of the first node, and the first data in the ferry pool are deleted.

[0279] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0280] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0281] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0282] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: For a first node, the method comprises: In the case that the second node fails, determining a plurality of first data to be cached and reconstructed included in a first log in the first node, where the plurality of first data are the same as a plurality of second data included in a second log in the second node; Writing each of the first data into the ferry pool in the first node; Output prompt information, where the prompt information is used to prompt that the cache rebuild is completed.

2. The method according to claim 1, characterized in that The step of writing each of the first data into a ferry pool in the memory of the first node includes: For each of the first data, mapping processing is performed on the first data according to a preset mapping method to obtain a first mapping address corresponding to the first data; The first data is written into the ferry pool according to the first mapping address.

3. The method according to claim 2, characterized in that The ferry pool includes a plurality of memory spaces, and writing the first data into the ferry pool according to the first mapping address includes: Matching the first mapping address with the mapping addresses corresponding to the memory spaces; The first data is written into the memory space corresponding to the successfully matched mapping address.

4. The method according to claim 1, characterized in that: The method further comprises: Receiving a write instruction sent by a host node, wherein the write instruction carries initial write data; When the transition memory cleanup time has not been reached, detecting whether the ferry pool is empty and obtaining a detection result; According to the detection result, the cache instance corresponding to the initial write data is written into the cache of the first node.

5. The method according to claim 4, characterized in that The step of writing the cache instance corresponding to the initial write data into the cache of the first node according to the detection result includes: If the detection result is that the ferry pool is not empty, mapping the initial written data according to a preset mapping method to obtain a second mapping address corresponding to the initial written data; Determine a target memory space corresponding to the second mapping address in the ferry pool according to the second mapping address, and match the initial write data with the target first data in the target memory space if the target memory space is not empty; If the initial write data matches the target first data, the target first data is deleted in the target memory space, and the cache instance corresponding to the initial write data is written into the cache of the first node.

6. The method according to claim 5, characterized in that Writing the cache instance corresponding to the initial write data into the cache of the first node includes: If the initial write data matches the target first data, indicating that the initial write data is completely identical to the target first data, generating the cache instance according to the initial write data, and writing the cache instance into the cache of the first node; If the initial write data matches the target first data, indicating that the initial write data is partially identical to the target first data, the initial write data is padded according to the portion of the target first data that is different from the initial write data to obtain the target write data, generate the cache instance according to the target write data, and write the cache instance into the cache of the first node.

7. The method according to claim 5, characterized in that The method further comprises: If the initial write data does not match the target first data, a cache instance is generated according to the initial write data, and the cache instance is written into the cache of the first node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images