Storage and compute decoupling system, data processing method and storage medium

Abstract

Provided in the embodiments of the present application are a storage and compute decoupling system, a data processing method and a nonvolatile readable storage medium. The system comprises: a memory cluster and a computing power cluster, the memory cluster comprising a first node, the computing power cluster comprising a fourth node, a second node and a third node belonging to nodes in the memory cluster and / or the computing power cluster, and M data blocks created by the fourth node being stored in the first node. The first node and / or the fourth node is used for determining the first quantity and the second quantity of data blocks, among the M data blocks, which are respectively subjected to a copying operation and an erasure coding operation; data obtained by copying the first quantity of data blocks is stored into the second node, and data obtained by performing an erasure coding operation on the second quantity of data blocks is stored into the third node; and among various combinations of values for the two quantities of data blocks subjected to the copying operation and data blocks subjected to the erasure coding operation, a resource consumption parameter corresponding to the first quantity and the second quantity is the minimum. The present application can improve the data processing efficiency.

Description

Storage and computing decoupling system, data processing method and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on January 31, 2024, with application number 2024101368527 and application name “Storage and Computing Decoupling System, Data Processing Method and Storage Medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of computers, and specifically, to a storage and computing decoupling system, a data processing method, and a non-volatile readable storage medium. Background Art

[0004] In a storage-computing decoupled system, since the applications running on the computing nodes rely on the random storage resources provided by the memory node cluster and are accessed through technologies such as Ethernet and PCIE (Peripheral Component Interconnect Express, a high-speed serial expansion bus standard) bus, the computing nodes are susceptible to various failure scenarios. Potential memory resource failures include link failures, independent or related failures of memory nodes, removal and damage of memory resources, etc. The occurrence of such problems will cause the computing system program that relies on remote memory nodes to crash, downtime, and other problems. In response to the various computing system crashes that may be caused by memory node failures, related technologies mainly use remote data multi-copy backup, memory data correction and deletion, and other technologies to redundancy of the node's memory data and improve system reliability.

[0005] However, memory replication consumes a large amount of memory and bandwidth resources, and erasure coding causes a loss of computing resources. This is especially true for nodes in remote memory clusters, where the CPU (Central Processing Unit) computing power is relatively low. Memory replication or memory erasure coding can lead to increased time and resource consumption for remote memory data recovery in decoupled computing systems, resulting in decreased availability.

[0006] Therefore, the data processing method in the related art has the problem of low data processing efficiency.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a storage-computing decoupling system, a data processing method, and a non-volatile readable storage medium to at least solve the problem of low data processing efficiency in the data processing method in the related art.

[0009] According to one embodiment of the present application, a storage-computing decoupling system is provided, comprising: a memory cluster and a computing power cluster, wherein the memory cluster comprises a first node, the computing power cluster comprises a fourth node, and the second node and the third node belong to the memory cluster and / or nodes in the computing power cluster; wherein the first node and / or the fourth node are used to determine a first number of data blocks to be copied and a second number of data blocks to be erased among M data blocks; data blocks obtained by copying the first number of data blocks among the M data blocks are stored in the second node, and data blocks obtained by erasing the second number of data blocks among the M data blocks are stored in the third node; wherein M is equal to the sum of the first number and the second number, and among various value combinations of the number of data blocks to be copied and the number of data blocks to be erased, resource consumption parameters corresponding to the first number and the second number are the smallest.

[0010] According to another embodiment of the present application, a data processing method is provided, comprising: upon determining that there are M data blocks to be backed up, determining a first number of data blocks to be copied and a second number of data blocks to be erased among the M data blocks, wherein M is a positive integer greater than or equal to 2, and M is equal to the sum of the first number and the second number, and among various value combinations of the number of data blocks to be copied and the number of data blocks to be erased, resource consumption parameters corresponding to the first number and the second number are minimized; performing a copy operation on a first number of data blocks among the M data blocks, and performing a first storage operation on data blocks obtained by performing the copy operation; performing an erase operation on a second number of data blocks among the M data blocks, and performing a second storage operation on data blocks obtained by performing the erase operation.

[0011] According to another embodiment of the present application, a computer non-volatile readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0012] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0013] Through the present application, a method of performing redundant processing on the backup data is adopted based on the data blocks to be copied and the data blocks to be erasure calculated in the data to be backed up when the determined resource consumption parameter is minimized. When it is determined that the data blocks to be backed up are M data blocks, a first number of data blocks to be copied and a second number of data blocks to be erased in the M data blocks are determined, wherein M is a positive integer greater than or equal to 2, and M is equal to the sum of the first number and the second number. Among various value combinations of the number of data blocks to be copied and the number of data blocks to be erased, the resource consumption parameters corresponding to the first number and the second number are minimized; the copy operation is performed on the first number of data blocks in the M data blocks. The method further comprises performing a first storage operation on the data blocks obtained by performing the copy operation; performing an erasure operation on a second number of the M data blocks, and performing a second storage operation on the data blocks obtained by performing the erasure operation. By determining the resource consumption parameter corresponding to the mixed and parallel multiple redundant processing and selecting the number of data blocks corresponding to the multiple redundant processings corresponding to the minimum resource consumption parameter, it is possible to ensure that the resource consumption required for redundant processing of the data blocks to be processed is minimized. At the same time, it is also possible to ensure that, in the event of a memory failure, the failed memory data can be recovered based on the data blocks stored by the redundant processing with minimal resource consumption, thereby reducing the resource consumption parameters required for data processing and data recovery and improving data processing efficiency. Therefore, it is possible to solve the problem of low data processing efficiency in the data processing methods of the related art and achieve the technical effect of improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a schematic diagram of a hardware environment of a data processing method according to an embodiment of the present application.

[0015] FIG2 is a flow chart of a data processing method according to an embodiment of the present application.

[0016] FIG3 is a schematic diagram of a data processing method according to an embodiment of the present application.

[0017] FIG4 is a schematic diagram of another data processing method according to an embodiment of the present application.

[0018] FIG5 is a flow chart of another data processing method according to an embodiment of the present application.

[0019] Figure 6 is a schematic diagram of a storage and computing decoupling system architecture according to an embodiment of the present application.

[0020] FIG7 is a schematic diagram of another storage-computing decoupling system architecture according to an embodiment of the present application.

[0021] FIG8 is a schematic diagram of another storage-computing decoupling system architecture according to an embodiment of the present application.

[0022] FIG9 is a schematic diagram of another storage-computing decoupling system architecture according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] It should be noted that the terms "target", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0025] The method embodiments provided in the embodiments of the present application can be executed in a node device or a similar computing device. Taking operation on a node device as an example, FIG1 is a hardware structure block diagram of a node device of a data processing method in an embodiment of the present application. As shown in FIG1 , the node device may include one or more (only one is shown in FIG1 ) processors 102 (the processor 102 may include but is not limited to a microprocessor MCU (Microcontroller Unit, micro control unit) or a programmable logic device FPGA (Field Programmable Gate Array, field programmable gate array) and a processing device) and a memory 104 configured to store data, wherein the above-mentioned node device may also include a transmission device 106 and an input and output device 108 configured to have a communication function. It can be understood by those skilled in the art that the structure shown in FIG1 is only for illustration and does not limit the structure of the above-mentioned node device. For example, the node device may also include more or fewer components than shown in FIG1 , or have a configuration different from that shown in FIG1 .

[0026] The memory 104 can be configured to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may include a memory remotely located relative to the processor 102, and these remote memories may be connected to the node device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0027] The transmission device 106 is configured to receive or transmit data via a network. Examples of the aforementioned network may include a wireless network provided by a communication provider of the node device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the transmission device 106 can be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0028] In this embodiment, a data processing method is provided. FIG2 is a flow chart of the data processing method according to the embodiment of the present application. As shown in FIG2 , the flow chart includes the following steps S202 to S206 .

[0029] Step S202: When it is determined that there are M data blocks to be backed up, determine a first number of data blocks to be copied and a second number of data blocks to be erased among the M data blocks, where M is a positive integer greater than or equal to 2, and M is equal to the sum of the first number and the second number. Among various combinations of the number of data blocks to be copied and the number of data blocks to be erased, the resource consumption parameters corresponding to the first number and the second number are minimum.

[0030] Step S204 : performing a copy operation on a first number of data blocks among the M data blocks, and performing a first storage operation on the data blocks obtained by performing the copy operation.

[0031] Step S206 : performing erasure correction operations on a second number of data blocks among the M data blocks, and performing a second storage operation on the data blocks obtained by performing the erasure correction operations.

[0032] The data processing method in this embodiment can be applied to the scenario of processing memory data in a storage-computing decoupling system. Here, the storage-computing decoupling system may refer to the decoupling of the two functional modules of storage and computing, and their separate deployment and management to improve the flexibility and reliability of the system. Inside this resource-decoupled and separated data center, there are nodes dedicated to processing specific tasks, including CPU computing nodes, memory nodes, storage nodes, etc. Compared with relatively mature resource decoupling technologies such as storage and computing, memory decoupling technology is in rapid development. As an important component of high-performance nodes, memory resources are highly coupled with computing resources such as the CPU. The CPU generally reads data from the memory through a dedicated on-chip bus link and has extremely strict latency requirements.

[0033] However, in a decoupled storage and computing system, applications running on compute nodes rely on random access memory resources provided by a cluster of memory nodes, accessed via technologies like Ethernet and the PCIe bus. These applications are susceptible to various failure scenarios. Potential memory resource failures include link failures, independent or correlated failures of memory nodes, and the removal and corruption of memory resources. Such issues can cause program crashes and downtime in computing systems that rely on remote memory nodes.

[0034] To address the various computing system crashes that can occur due to failures, related technologies typically implement redundant node memory data through remote data replication and memory data erasure, thereby improving system reliability. However, memory replication consumes significant memory and bandwidth resources, and erasure coding can lead to a loss of computing resources. This is particularly true given the relatively low CPU computing power of nodes in remote memory clusters. These factors increase the time and resource consumption of remote memory data recovery in decoupled computing systems, leading to decreased availability.

[0035] To at least address some of the above-mentioned issues, in this embodiment, for memory data in a storage-computation decoupling system, in order to avoid data corruption caused by failures, redundant processing can be performed on the data after it is created. Before performing the redundant processing, the number of data blocks currently to be backed up can be determined, and the number of data blocks to be copied and the number of data blocks to be erased can be determined, among various combinations of values, when the resource consumption parameter is minimized, i.e., the first number and the second number in this embodiment. The redundant processing in this embodiment can include at least one of a copy operation and an erasure operation.

[0036] It should be noted that the above-mentioned resource consumption parameters can be used to indicate the time consumed for performing the copy operation and the corresponding first storage operation on the data block to be copied, the storage space occupied by the data block to be copied, and the time consumed for performing the erasure operation and the corresponding second storage operation on the data block to be erased, and the storage space occupied by the data block to be erased.

[0037] In this embodiment, the copy operation may refer to copying multiple data blocks from the M data blocks, and the first storage operation may refer to transferring the copied copies to other physical nodes in the storage-computing decoupling system for storage. The erasure operation may refer to performing erasure correction calculations on multiple data blocks from the M data blocks, and the second storage operation may refer to transferring the calculated erasure blocks to other physical nodes in the storage-computing decoupling system for storage.

[0038] As shown in Figure 3, the M data blocks on the first node are divided into two groups of data blocks. The first group of data blocks (the first number of data blocks) are copied to obtain copies and transmitted to the second node. The second group of data blocks (the second number of data blocks) undergo erasure calculation to obtain erasure blocks and transmit them to the third node.

[0039] It should be noted that the above-mentioned erasure correction operation may refer to encoding calculations on a second number of data blocks among the M data blocks to obtain K erasure correction blocks. When the second number of data blocks fail, the erasure correction blocks can be used to restore the failed second number of data blocks through the corresponding reconstruction algorithm.

[0040] The above-mentioned encoding calculation can be a linear algebra operation. During encoding, based on the first calculation parameter and the second calculation parameter, a first matrix is ​​constructed (the rows of the matrix are determined by the sum of the first calculation parameter and the second calculation parameter, and the columns of the matrix are determined by the first calculation parameter), and the second number of data blocks are converted into a first column vector. The first matrix is ​​multiplied by the first column vector to obtain a new column vector. The new column vector includes the erasure blocks obtained by this calculation. Correspondingly, the reconstruction algorithm, that is, the data blocks remaining after the failure of the second number of data blocks are combined with the erasure blocks to form a second column vector. The second column vector is multiplied by the second matrix (the inverse matrix of the first matrix) to obtain the first column vector corresponding to the second number of data blocks, thereby completing the recovery of the second number of data blocks. Here, the first calculation parameter can be the numerical value corresponding to the second number, and the second calculation parameter can be the numerical value of the specified number of erasure blocks.

[0041] For example, as shown in Figure 4, the first calculation parameter is 5, the second calculation parameter is 3, the first matrix is ​​matrix A (8 rows, 5 columns), and the first column vector is column vector D. In the erasure operation, matrix B is multiplied by column vector D, and the product obtained is the information column vector D+C, where C corresponds to the erasure data block.

[0042] Optionally, after determining the first and second quantities corresponding to the minimum resource consumption parameter among various value combinations, the determined first and second quantities may be adjusted based on a preset parameter, with the sum of the adjusted first and second quantities remaining M. The preset parameter may be determined based on historical partitioning results, where the historical partitioning results may be the ratio of the number of data blocks corresponding to different redundant processing methods selected when the same parallel redundant processing method is applied to different memory data before redundant processing is performed on the aforementioned data blocks to be processed, to the total number of data blocks. Furthermore, the preset parameter may be determined based on the success rate of multiple historical redundant processing operations.

[0043] Through the above steps S202 to S206, when it is determined that there are M data blocks to be backed up, a first number of data blocks to be copied and a second number of data blocks to be erased are determined among the M data blocks, where M is a positive integer greater than or equal to 2, and M is equal to the sum of the first number and the second number. Among various combinations of the number of data blocks to be copied and the number of data blocks to be erased, the resource consumption parameters corresponding to the first number and the second number are minimized. The copy operation is performed on the first number of data blocks among the M data blocks, and the first storage operation is performed on the data blocks obtained by performing the copy operation. The erasure operation is performed on the second number of data blocks among the M data blocks, and the second storage operation is performed on the data blocks obtained by performing the erasure operation. This can reduce data redundancy costs while ensuring the reliability of memory cluster data, and solve the problem of low data processing efficiency in data processing methods in related technologies.

[0044] The execution subject of the above steps may be a node, a terminal, etc., but is not limited thereto.

[0045] In an exemplary embodiment, determining that the number of data blocks to be copied among the M data blocks is a first number and the number of data blocks to be erased is a second number includes steps S11 to S12.

[0046] S11, when the number of data blocks to be copied among M data blocks is a first variable and the number of data blocks to be erased is a second variable, determine a target expression for a resource consumption parameter based on the first variable and the second variable, wherein the value of the target expression represents the resource consumption parameter.

[0047] S12, when the value of the target expression is minimum when the value of the first variable is the first number and the value of the second variable is the second number, the number of data blocks to be copied in the M data blocks is determined as the first number, and the number of data blocks to be erased in the M data blocks is determined as the second number.

[0048] To determine the first and second quantities corresponding to the minimum resource consumption parameter, in this embodiment, a target expression for the resource consumption parameter can be constructed based on the time and space costs of the copy operation and the corresponding storage operation, as well as the time and space costs of the erasure operation and the corresponding storage operation. In the target expression, the variables are a first variable representing the number of data blocks corresponding to the copy operation and a second variable representing the number of data blocks corresponding to the erasure operation.

[0049] As the values ​​of the first variable and the second variable change, the value of the target expression may also change. When the value of the first variable is the first quantity and the value of the second variable is the second quantity, the value of the target expression is minimum, indicating that the value of the resource consumption parameter is minimum.

[0050] Through this embodiment, based on the constructed expression with the number of data blocks corresponding to the two redundant processing as a variable, the number of data blocks corresponding to the two redundant processing when the resource consumption parameter is minimized is determined. This can determine a data redundancy strategy with the lowest overall cost for memory data and ensure that data can be accurately and quickly restored when a failure occurs.

[0051] In an exemplary embodiment, before determining the number of data blocks to be copied among the M data blocks as a first number and determining the number of data blocks to be erased among the M data blocks as a second number, the method further includes steps S21 to S22.

[0052] S21. When various value combinations of the number of data blocks to be copied and the number of data blocks to be erased include N value combinations, and each value combination in the N value combinations includes the value of the first variable and the value of the second variable, when the number of data blocks to be copied and the number of data blocks to be erased are each value combination in the N value combinations, determine the value of the target expression, and obtain a total of N values ​​of the target expression, where N is a positive integer greater than or equal to 2.

[0053] S22, determine the minimum value among N values, and determine the value of the first variable in a value combination corresponding to the minimum value as the first quantity, and determine the value of the second variable in a value combination as the second quantity, wherein the minimum value is the value of the target expression when the number of data blocks to be copied and the number of data blocks to be erased are a value combination.

[0054] In this embodiment, for the values ​​of the first and second variables when the target expression is minimized, since the sum of the values ​​of the first and second variables must be equal to M, various combinations of values ​​for the number of data blocks to be replicated and the number of data blocks to be erased can be pre-determined. The values ​​of the first and second variables in each combination are substituted into the target expression to calculate the value of the resource consumption parameter corresponding to each combination. N combinations of values ​​correspond to N values ​​of the resource consumption parameter.

[0055] The value with the smallest value may be directly selected from the N values, and the two values ​​in the value combination corresponding to the value may be used as the first quantity and the second quantity.

[0056] Optionally, before determining the number of data blocks to be copied among the M data blocks as the first number, and determining the number of data blocks to be erased among the M data blocks as the second number, the above method also includes: solving the minimum value of the target expression within the value range of the first variable and the value range of the second variable, obtaining the value of the first variable and the value of the second variable when the value of the target expression is the minimum value, and determining the value of the first variable when the value of the target expression is the minimum value as the first number, and determining the value of the second variable when the value of the target expression is the minimum value as the second number, wherein the minimum value is the value of the target expression when the value of the first variable is the first number and the value of the second variable is the second number.

[0057] Considering that the larger M is, the more value combinations there are, in order to shorten the calculation time, in this embodiment, when determining the first quantity and the second quantity, the target expression can be solved, that is, the target expression can be regarded as an objective function, and by solving the objective function, the values ​​of the first variable and the second variable (that is, the first quantity and the second quantity) when the value of the target expression is minimum can be obtained.

[0058] In the objective function, the value range of the first variable and the value range of the second variable can be set. The solution of the objective function can be regarded as a mixed integer nonlinear programming problem, which is an NP-hard (Non-deterministic Polynomial-hard) problem. It can be approximately solved by using commercial solvers, branch and bound methods, etc., which will not be described in detail in this embodiment.

[0059] Through this embodiment, when the number of data blocks to be backed up is small, the value combination can be determined first, and then the value of the target expression corresponding to each value combination can be calculated, so as to select the values ​​of the first variable and the second variable when the expression value is minimum, thereby ensuring the accuracy of the calculation. When the number of data blocks to be backed up is large, the target expression can be solved directly, and the values ​​of the first variable and the second variable when the expression value is minimum can be determined according to the solved numerical value, thereby ensuring the calculation efficiency.

[0060] In an exemplary embodiment, determining a target expression for a resource consumption parameter according to a first variable and a second variable includes steps S31 to S33 .

[0061] S31, determine a first expression based on the first variable, and determine a second expression based on the first variable, wherein the first expression is used to represent the time consumed to perform the copy operation and the first storage operation on the data block to be copied, and the second expression is used to represent the storage space occupied by the data block to be copied.

[0062] S32. Determine a third expression based on the second variable and the third variable, and determine a fourth expression based on the third variable, wherein the third expression is used to represent the time consumed by performing the erasure operation and the second storage operation on the data blocks to be subjected to the erasure operation, the fourth expression is used to represent the storage space occupied by the data blocks to be subjected to the erasure operation, and the third variable is used to represent the number of data blocks obtained by performing the erasure operation on the data blocks whose number is the second variable.

[0063] S33, determining a target expression according to the first expression, the second expression, the third expression and the fourth expression.

[0064] In this embodiment, the target expression may be determined by multiple expressions, and the multiple expressions may respectively include an expression corresponding to the copy operation and an expression corresponding to the erasure operation.

[0065] Since the copy operation includes a copy operation and a first storage operation, the expression corresponding to the copy operation may include an expression corresponding to the time consumed by performing the copy operation and the storage operation on the data block to be copied, and the storage space occupied by the data block to be copied.

[0066] Correspondingly, since the erasure operation includes the erasure operation and the first storage operation, the expression corresponding to the erasure operation may include an expression corresponding to the time consumed by performing the erasure operation and the storage operation on the data block to be subjected to the erasure operation, the storage space occupied by the data block to be subjected to the erasure operation, and the number of data blocks obtained by performing the erasure operation on the data blocks whose number is a second variable.

[0067] Through this embodiment, expressions corresponding to different redundant processes are first determined respectively, and then based on the multiple expressions, the target expression corresponding to the parallel redundancy strategy is obtained, which can improve the efficiency of determining the target expression.

[0068] In an exemplary embodiment, determining a first expression according to a first variable includes step S41 .

[0069] S41, determining a first expression based on a first variable, a target size of the data block and a first transmission speed, wherein the target size of the data block is an average size of M data blocks or the size of one data block among the M data blocks, the first transmission speed is a speed for transmitting data between a node where a data block obtained by performing a copy operation is located and a second node, and the first storage operation is used to store the data block obtained by performing the copy operation on the second node.

[0070] Optionally, determining a first expression according to the first variable, the target size of the data block and the first transmission speed includes: determining the first expression as Wherein, dtr(Mr)=Mr*bs / B1, Mr represents the first variable, bs represents the target size of the data block, and B1 represents the first transmission speed.

[0071] It should be noted that B1 represents the data transmission capacity of the link between two physical nodes in the memory cluster (i.e., the first transmission speed mentioned above). dtr(Mr) represents the data transmission time required to transfer data from the current node where the data block obtained by the copy operation is located to the second node. The first expression above maps the data transmission time to the interval [0, 1].

[0072] Optionally, determining the second expression based on the first variable includes: determining the second expression based on the first variable and a target remaining number, wherein the target remaining number is used to represent an average value of a maximum number of data blocks allowed to be stored in multiple nodes.

[0073] It should be noted that the above target remaining quantity, that is, the average maximum remaining data block storage space of each node in the memory cluster, represents the current available storage space level in the entire system.

[0074] Optionally, determining a second expression based on the first variable and the target remaining quantity includes: determining the second expression as Among them, Mr represents the first variable, Indicates the target remaining quantity.

[0075] It should be noted that the sum of the first expression and the second expression can be the resource consumption parameter corresponding to the copy operation, that is,

[0076] Through the first expression and the second expression, the resource consumption parameters corresponding to the copy operation can be determined, which are related to the transmission capacity between physical nodes and the data space occupancy ratio. The smaller the value of the resource consumption parameter, the lower the time and space cost of copying the current data.

[0077] In an exemplary embodiment, determining a third expression according to the second variable and the third variable includes step S51 .

[0078] S51. Determine a third expression based on the second variable, the third variable, the size of each data block in the data blocks obtained by performing the erasure operation on the data blocks having the second variable number, the second transmission speed, and the target processing frequency, wherein the second transmission speed is the speed of transmitting data between the node where the data block obtained by performing the erasure operation is located and the third node, the second storage operation is used to store the data block obtained by performing the erasure operation on the third node, and the target processing frequency is used to represent the frequency at which the processor on the node where the data block obtained by performing the erasure operation is located processes data.

[0079] It should be noted that the second transmission speed is similar to the description of the first transmission speed, and will not be described in detail in this embodiment. The target processing frequency may represent the computing power of the CPU.

[0080] Optionally, determining a third expression based on the second variable, the third variable, the size of each data block in the data blocks obtained by performing the erasure operation on the data blocks having the second variable number, the second transmission speed, and the target processing frequency includes: determining the third expression as in, d cal (M e )=W*M e / fepu, wherein Me represents the second variable, K represents the third variable, bsk represents the size of the kth data block in the data blocks obtained by performing the erasure operation on the data blocks having the second variable number, B2 represents the second transmission speed, fcpu represents the target processing frequency, and W is a preset coefficient.

[0081] It should be noted that the above dcal(Me) represents the delay generated when the data block Me is subjected to erasure processing on a node with a computing power of fcpu, and W is a preset coefficient for processing a unit data block. The third expression above means that Mapped to the interval [0,1].

[0082] Optionally, determining a fourth expression according to the third variable includes:

[0083] A fourth expression is determined according to the third variable and the target remaining number, wherein the target remaining number is used to represent an average value of a maximum number of data blocks allowed to be stored in the plurality of nodes.

[0084] Optionally, determining a fourth expression based on the third variable and the target remaining quantity includes: determining the fourth expression as Among them, K represents the third variable, Indicates the target remaining quantity.

[0085] It should be noted that the sum of the third expression and the fourth expression can be the resource consumption parameter corresponding to the erasure operation, that is,

[0086] Through the third and fourth expressions, the resource consumption parameters corresponding to the erasure operation can be determined, which are not only related to the computational and transmission time costs generated by the erasure strategy, but also to the data amplification scale (i.e., the number of data blocks obtained by the erasure operation).

[0087] Optionally, determining a target expression based on the first expression, the second expression, the third expression and the fourth expression includes: determining the target expression to be equal to an expression obtained by performing an addition operation on the first expression, the second expression, the third expression and the fourth expression.

[0088] In this embodiment, the target expression is

[0089] Correspondingly, the objective function in the above embodiment can be

[0090] In an exemplary embodiment, when M data blocks are in a first node, a copy operation is performed on a first number of data blocks in the M data blocks, and a first storage operation is performed on the data blocks obtained by performing the copy operation, including: the first node performs a copy operation on the first number of data blocks in the M data blocks, and stores the data blocks obtained by performing the copy operation in a second node.

[0091] Performing an erasure operation on a second number of data blocks among the M data blocks, and performing a second storage operation on the data blocks obtained by performing the erasure operation, including: performing an erasure operation on the second number of data blocks among the M data blocks by a first node, and storing the data blocks obtained by performing the erasure operation in a third node.

[0092] The second node is different from the first node, and the third node is different from the first node.

[0093] The various nodes in this embodiment can be nodes in a storage-computing decoupling system, and a node can refer to a server. The storage-computing decoupling system can be composed of a computing power cluster and a memory cluster. That is, the storage-computing decoupling system can include a memory server cluster and a computing power server cluster. The servers in the memory server cluster are primarily configured to store data, while the servers in the computing power server cluster are primarily configured to perform data processing and corresponding computing operations. However, for computing operations with smaller computational loads, servers in the memory server cluster can also be used to perform them.

[0094] Therefore, in this embodiment, for the M data blocks on the first node, the first and second quantities can be determined by a server with greater computing power (e.g., the fourth node). The replication operation and its first storage operation, as well as the computationally less computationally intensive erasure operation and its second storage operation, can be performed by the first node storing the M data blocks.

[0095] Optionally, when there are M data blocks in the first node, a copy operation is performed on a first number of data blocks among the M data blocks, and a first storage operation is performed on the data blocks obtained by performing the copy operation, including: the first node performs a copy operation on the first number of data blocks among the M data blocks, and stores the data blocks obtained by performing the copy operation in the second node.

[0096] Performing an erasure operation on a second number of data blocks among the M data blocks, and performing a second storage operation on the data blocks obtained by performing the erasure operation, including: obtaining the second number of data blocks among the M data blocks by a fourth node, performing the erasure operation on the second number of data blocks among the M data blocks, and storing the data blocks obtained by performing the erasure operation in a third node; wherein the second node is different from the first node, and the third node is different from the first node.

[0097] Optionally, when there are M data blocks in the first node, a copy operation is performed on a first number of data blocks among the M data blocks, and a first storage operation is performed on the data blocks obtained by performing the copy operation, including: obtaining M data blocks by a fourth node, performing a copy operation on the first number of data blocks among the M data blocks, and storing the data blocks obtained by performing the copy operation in the second node.

[0098] Performing an erasure operation on a second number of data blocks among the M data blocks, and performing a second storage operation on the data blocks obtained by performing the erasure operation, including: performing the erasure operation on the second number of data blocks among the M data blocks by a fourth node, and storing the data blocks obtained by performing the erasure operation in a third node; wherein the second node is different from the first node, and the third node is different from the first node.

[0099] In this embodiment, the copy operation and its first storage operation, and / or the erasure operation with less computational complexity and its second storage operation, can also be performed by the fourth node in the computing power server cluster, but the fourth node needs to first obtain the corresponding data block from the first node before it can perform the corresponding operation.

[0100] It should be noted that when the fourth node performs the copy operation and its first storage operation, and / or the erasure operation with less computational complexity and its second storage operation, the time and space cost required to transmit the data block from the first node to the fourth node can also be included in the calculation of the comprehensive cost.

[0101] It should be noted that the second node can serve as either a compute node or a memory node in the storage-computing decoupling system. That is, the copy obtained by the replication operation can be stored in either a compute node or a memory node. The second node and the first node can be physically separate nodes, serving as two physical nodes in the storage-computing decoupling system.

[0102] In this embodiment, the third node storing the erasure block may be a node different from the aforementioned second node, or may be the same node as the second node.

[0103] The third node can serve as a computing node in the storage-computing decoupling system, or as a memory node in the storage-computing decoupling system. That is, the erasure block obtained by the erasure operation can be stored in the computing node or in the memory node. The erasure operation and the second storage operation can also be performed by the first node. Since the third node can be different from the first node (that is, the third node can be a node physically separated from the first node), the erasure block can be processed at a computing node and stored in another computing node, or processed at a memory node and stored in a computing node, or processed at a computing node and stored in a memory node, or processed at a memory node and stored in another memory node.

[0104] Optionally, since erasure operations, unlike copy operations, involve computation, if the first node is not a computation node, the node performing the erasure operation can be a fourth node, which is also a computation node. However, because the M data blocks are stored in the first node, before performing the erasure operation, the fourth node must first obtain the data block to be erased from the M data blocks. Accordingly, the erasure block can be transmitted from the fourth node to the third node.

[0105] Through this embodiment, the data blocks obtained by redundant processing are stored in another node, which can prevent the node where the original data block is located from being affected by the failure and provide a basis for recovering the damaged data block, thereby ensuring the normal operation of the computing system program.

[0106] In an exemplary embodiment, before determining that the number of data blocks to be copied among the M data blocks is a first number and the number of data blocks to be erased is a second number, the method further includes step S61.

[0107] S61, when M data blocks are created in the memory space of the first node, it is determined that the data blocks to be backed up are M data blocks, wherein the M data blocks are created by the fourth node and the memory space of the first node is the remote memory space of the fourth node.

[0108] In this embodiment, the memory space of the first node can also be created by the fourth node. The fourth node serves as a computing node in the storage and computing decoupling system, and the first node serves as a memory node in the storage and computing decoupling system. Correspondingly, the M data blocks can also be stored by the fourth node in the memory space of the first node.

[0109] Optionally, the memory space of the first node may be created by the first node, and the M data blocks may be stored by the first node.

[0110] Through this embodiment, the data blocks to be backed up can be created and stored in the memory node, or created and stored in the memory node by the computing node. This can ensure that in the storage and computing decoupling system, all memory data can be processed redundantly, thereby ensuring the reliability of memory resource access.

[0111] The data processing method in the embodiment of the present application is explained below with reference to optional examples. This optional example provides a fault handling method for a storage-computing decoupling system. In response to the problems of high data recovery cost, low time and resource efficiency when memory failure occurs in the existing storage-computing decoupling architecture, a fault handling architecture with multiple redundant strategies in parallel is constructed from the perspective of performance and cost balance. By modeling the implementation costs of different fault recovery strategies and constructing a hybrid parallel fault recovery mechanism for memory data, the efficiency of remote memory resource fault handling can be effectively improved, and the fault recovery time and resource consumption can be reduced.

[0112] The process of the data processing method in this optional example can be shown in Figure 5.

[0113] Step 1: Count the size of target data that needs to be redundant.

[0114] Step 2: Calculate the combined cost of replication and erasure.

[0115] The comprehensive cost of the replication strategy Rrep(M) is The comprehensive cost of the replication strategy Rrep is related to the transmission capacity between physical nodes and the size of the data space occupied;

[0116] The erasure strategy expands the data block to form M+K data blocks, where K data blocks are new data generated based on calculations and sent to physically separated nodes. The corresponding comprehensive cost Rera(M,K) is calculated as follows: Rera is not only related to the computational and transmission time costs generated by the erasure strategy, but also to the data amplification scale K.

[0117] When the size of the target data to be redundant is M data blocks, the number of data blocks to be copied is Mr, and the number of data blocks to be erased is Me. Then the cost-benefit of multi-dimensional redundancy for M blocks of data is R mix =R rep (M r )+R era (M e ,,K), that is, the comprehensive cost is

[0118] Step 3: Split the target data into two parts.

[0119] Step 4: Perform replica replication and erasure calculation respectively, and transfer the corresponding data to other physical nodes for storage.

[0120] When local data fails, the replica data is sent back to the local area for recovery. At the same time, the erasure data is calculated and the lost data is restored to achieve complete data recovery capabilities.

[0121] This optional example can provide a highly reliable and low-cost data redundancy strategy generation framework for large-scale storage and computing decoupling systems, promoting the decoupling of resources such as memory and computing power.

[0122] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are optional embodiments for implementing the scheme of this application, and the actions and modules involved are not necessarily required for this application.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a non-volatile readable storage medium (such as ROM / RAM, disk, CD-ROM), including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, node, or network device, etc.) to execute the methods of each embodiment of the present application.

[0124] This embodiment further provides a storage and computing decoupling system, which is configured to implement the above embodiment and optional implementations. Details already described will not be repeated here. The system includes:

[0125] A memory cluster and a computing power cluster, wherein the memory cluster includes a first node, the computing power cluster includes a fourth node, and the second node and the third node belong to the memory cluster and / or nodes in the computing power cluster; wherein the first node and / or the fourth node are used to determine a first number of data blocks to be copied and a second number of data blocks to be erased among M data blocks; the data blocks obtained by copying the first number of data blocks among the M data blocks are stored in the second node, and the data blocks obtained by erasing the second number of data blocks among the M data blocks are stored in the third node; wherein M is equal to the sum of the first number and the second number, and among various value combinations of the number of data blocks to be copied and the number of data blocks to be erased, the resource consumption parameters corresponding to the first number and the second number are the smallest.

[0126] Taking the cluster as a server cluster and the node as a server as an example, as shown in Figure 6, the fourth node is a server in the computing power server cluster and can access the corresponding remote memory space in the memory server cluster. The first node can be a server in the memory server cluster, that is, the remote memory space accessed by the fourth node can be located on the first node. The memory data (M data blocks) in the remote memory space can be divided into two parts. One part of the memory data (i.e., the first number of data blocks) is copied to the second node, and the other part of the memory data (i.e., the second number of data blocks) is processed by erasure calculation to obtain K erasure blocks. The K erasure blocks can be transmitted to the third node. The second and third nodes can be servers in the memory server cluster or servers in the computing power server cluster. Correspondingly, after the memory data transmission failure on the first node, the replica and / or erasure blocks can be directly obtained from the second and third nodes, thereby realizing the recovery of the memory data.

[0127] In an exemplary embodiment, a fourth node is used to determine a first number of data blocks to be copied and a second number of data blocks to be erased among M data blocks, and send the first number and the second number to the first node; the first node is used to perform a copy operation on a first number of data blocks among the M data blocks, and store the data blocks obtained by performing the copy operation to the second node; and perform an erase operation on a second number of data blocks among the M data blocks, and store the data blocks obtained by performing the erase operation to the third node, wherein the second node and the third node are the same node, or the second node and the third node are different nodes.

[0128] Taking a server cluster as an example and a server as a node, as shown in Figure 7, the fourth node can calculate and determine the number of data blocks in each part (i.e., a first number and a second number) when the memory data (M data blocks) in its remote memory space needs to be divided into two parts, and transmit the calculations to the first node. The first node copies the first number of the M data blocks to the second node, performs erasure calculations on the second number of the M data blocks to obtain K erasure blocks, and transmits the K erasure blocks to the third node.

[0129] In an exemplary embodiment, the fourth node is used to determine a first number of data blocks to be copied and a second number of data blocks to be erased among M data blocks, and send the first number to the first node; the first node is used to perform a copy operation on the first number of data blocks among the M data blocks, and store the data blocks obtained by performing the copy operation to the second node; the fourth node is also used to obtain a second number of data blocks among the M data blocks, perform an erase operation on the second number of data blocks among the M data blocks, and store the data blocks obtained by performing the erase operation to the third node; wherein the second node and the third node are the same node, or the second node and the third node are different nodes.

[0130] Taking the cluster as a server cluster and the node as a server as an example, as shown in Figure 8, the fourth node can determine through calculation that when the memory data (M data blocks) in its remote memory space needs to be divided into two parts, the number of data blocks in each part (i.e., the first number and the second number). Since the copy operation does not involve calculation processing, and the erasure operation requires calculation, the first node can copy the first number of memory data in the M data blocks to the second node, but the fourth node obtains the second number of memory data in the M data blocks from the first node, and performs erasure calculation processing on it to obtain K erasure blocks, and transmit the K erasure blocks to the third node.

[0131] In an exemplary embodiment, a fourth node is used to determine a first number of data blocks to be copied and a second number of data blocks to be erased among M data blocks; the fourth node is also used to obtain M data blocks from the first node; perform a copy operation on a first number of data blocks among the M data blocks, and store the data blocks obtained by performing the copy operation to the second node; perform an erase operation on a second number of data blocks among the M data blocks, and store the data blocks obtained by performing the erase operation to the third node, wherein the second node and the third node are the same node, or the second node and the third node are different nodes.

[0132] Taking the cluster as a server cluster and the nodes as servers as an example, as shown in Figure 9, the fourth node can calculate and determine the number of data blocks in each part (i.e., the first number and the second number) when the memory data (M data blocks) in its remote memory space needs to be divided into two parts. The fourth node then copies the first number of memory data in the M data blocks to the second node, performs erasure calculations on the second number of memory data in the M data blocks, obtains K erasure blocks, and transmits the K erasure blocks to the third node. However, the fourth node must first obtain the M data blocks from the first node before it can begin operations such as copying and erasure.

[0133] In an exemplary embodiment, the fourth node is also used to determine a target expression for a resource consumption parameter based on the first variable and the second variable, when the number of data blocks to be copied in the M data blocks is a first variable and the number of data blocks to be erased is a second variable, wherein the value of the target expression represents the resource consumption parameter; and when the value of the target expression is minimum when the value of the first variable is a first number and the value of the second variable is a second number, the number of data blocks to be copied in the M data blocks is determined to be a first number, and the number of data blocks to be erased in the M data blocks is determined to be a second number.

[0134] In an exemplary embodiment, the fourth node is further configured to, before determining the number of data blocks to be copied among the M data blocks as the first number and the number of data blocks to be erased among the M data blocks as the second number, determine a value of a target expression when the number of data blocks to be copied and the number of data blocks to be erased are each value combination in the N value combinations, obtaining a total of N values ​​of the target expression, where N is a positive integer greater than or equal to 2; and determine a minimum value among the N values, and determine a value combination corresponding to the minimum value that includes the value of the first variable as the first number, and a value combination that includes the value of the second variable as the second number, where the minimum value is the value of the target expression when the number of data blocks to be copied and the number of data blocks to be erased are each value combination in the N value combinations.

[0135] In an exemplary embodiment, the fourth node is also used to solve the minimum value of the target expression within the value range of the first variable and the value range of the second variable before determining the number of data blocks to be copied among the M data blocks as the first number and the number of data blocks to be erased among the M data blocks as the second number, to obtain the value of the first variable and the value of the second variable when the value of the target expression is the minimum value, and determine the value of the first variable as the first number when the value of the target expression is the minimum value, and determine the value of the second variable as the second number when the value of the target expression is the minimum value, wherein the minimum value is the value of the target expression when the value of the first variable is the first number and the value of the second variable is the second number.

[0136] In an exemplary embodiment, the fourth node is further used to determine a first expression based on the first variable, and to determine a second expression based on the first variable, wherein the first expression is used to represent the time consumed by performing a copy operation and a first storage operation on a data block to be copied, and the second expression is used to represent the storage space occupied by the data block to be copied; determine a third expression based on the second variable and the third variable, and determine a fourth expression based on the third variable, wherein the third expression is used to represent the time consumed by performing an erasure operation and a second storage operation on a data block to be erased, the fourth expression is used to represent the storage space occupied by the data block to be erased, and the third variable is used to represent the number of data blocks obtained by performing an erasure operation on data blocks whose number is the second variable; and determine a target expression based on the first expression, the second expression, the third expression, and the fourth expression.

[0137] In an exemplary embodiment, the fourth node is further used to determine a first expression based on the first variable, the target size of the data block and the first transmission speed, wherein the target size of the data block is the average size of M data blocks or the size of one data block among the M data blocks, the first transmission speed is the speed of transmitting data between the node where the data block obtained by performing the copy operation is located and the second node, and the first storage operation is used to store the data block obtained by performing the copy operation on the second node.

[0138] In an exemplary embodiment, the fourth node is further configured to determine the first expression as Wherein, dtr(Mr)=Mr*bs / B1, Mr represents the first variable, bs represents the target size of the data block, and B1 represents the first transmission speed.

[0139] In an exemplary embodiment, the fourth node is further configured to determine a second expression based on the first variable and a target remaining number, wherein the target remaining number is configured to represent an average value of a maximum number of data blocks allowed to be stored in the plurality of nodes.

[0140] In an exemplary embodiment, the fourth node is further configured to determine the second expression as Among them, Mr represents the first variable, Indicates the target remaining quantity.

[0141] In an exemplary embodiment, the fourth node is further used to determine a third expression based on the second variable, the third variable, the size of each data block in the data blocks obtained by performing an erasure operation on the data blocks having the second variable number, the second transmission speed, and the target processing frequency, wherein the second transmission speed is the speed of transmitting data between the node where the data blocks obtained by performing the erasure operation are located and the third node, the second storage operation is used to store the data blocks obtained by performing the erasure operation on the third node, and the target processing frequency is used to represent the frequency at which the processor on the node where the data blocks obtained by performing the erasure operation are located processes data.

[0142] In an exemplary embodiment, the fourth node is further configured to determine the third expression as in, d cal (M e )=W*M e / fepu, wherein Me represents the second variable, K represents the third variable, bsk represents the size of the kth data block in the data blocks obtained by performing the erasure operation on the data blocks having the second variable number, B2 represents the second transmission speed, fcpu represents the target processing frequency, and W is a preset coefficient.

[0143] In an exemplary embodiment, the fourth node is further configured to determine a fourth expression based on the third variable and the target remaining number, wherein the target remaining number is configured to represent an average value of a maximum number of data blocks allowed to be stored in the plurality of nodes.

[0144] In an exemplary embodiment, the fourth node is further configured to determine the fourth expression as Among them, K represents the third variable, Indicates the target remaining quantity.

[0145] In an exemplary embodiment, the fourth node is further configured to determine the target expression to be equal to an expression obtained by performing an addition operation on the first expression, the second expression, the third expression, and the fourth expression.

[0146] In an exemplary embodiment, the first node is further used to determine that the data blocks to be backed up are M data blocks when M data blocks are created in the memory space of the first node before determining that the number of data blocks to be copied among the M data blocks is a first number and the number of data blocks to be erased is a second number, wherein the M data blocks are created by the fourth node and the memory space of the first node is the remote memory space of the fourth node.

[0147] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0148] An embodiment of the present application further provides a computer non-volatile readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0149] In an exemplary embodiment, the above-mentioned computer non-volatile readable storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0150] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0151] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0152] The examples in this embodiment can refer to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0153] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0154] The above are merely optional embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may be subject to various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A storage and computing decoupling system, characterized in that: include: A memory cluster and a computing power cluster, wherein the memory cluster includes a first node, the computing power cluster includes a fourth node, the second node and the third node belong to the memory cluster, and / or the first node in the computing power cluster stores M data blocks created by the fourth node; The first node and / or the fourth node is configured to determine a first number of data blocks to be copied and a second number of data blocks to be erased from the M data blocks; store data blocks obtained by copying the first number of data blocks in the M data blocks to the second node, and store data blocks obtained by erasing the second number of data blocks in the M data blocks to the third node; Here, M is equal to the sum of the first number and the second number, and among various combinations of the number of data blocks to be copied and the number of data blocks to be erased, the resource consumption parameters corresponding to the first number and the second number are the smallest.

2. The system according to claim 1, wherein: the fourth node is configured to determine a first number of data blocks to be copied and a second number of data blocks to be erased among the M data blocks, and send the first number and the second number to the first node; The first node is used to perform a copy operation on the first number of data blocks in the M data blocks, and store the data blocks obtained by performing the copy operation in the second node; perform an erasure operation on the second number of data blocks in the M data blocks, and store the data blocks obtained by performing the erasure operation in the third node, wherein the second node and the third node are the same node, or the second node and the third node are different nodes.

3. The system according to claim 1, wherein: the fourth node is configured to determine a first number of data blocks to be copied and a second number of data blocks to be erased among the M data blocks, and send the first number to the first node; The first node is configured to perform a copy operation on the first number of data blocks among the M data blocks, and store the data blocks obtained by performing the copy operation in the second node; The fourth node is further configured to obtain the second number of data blocks from the M data blocks, perform an erasure operation on the second number of data blocks from the M data blocks, and store the data blocks obtained by performing the erasure operation in the third node; The second node and the third node are the same node, or the second node and the third node are different nodes.

4. The system according to claim 1, wherein: The fourth node is configured to determine a first number of data blocks to be copied and a second number of data blocks to be erased, among the M data blocks; The fourth node is further configured to obtain the M data blocks from the first node; Perform a copy operation on the first number of data blocks in the M data blocks, and store the data blocks obtained by performing the copy operation in the second node; perform an erasure operation on the second number of data blocks in the M data blocks, and store the data blocks obtained by performing the erasure operation in the third node, wherein the second node and the third node are the same node, or the second node and the third node are different nodes.

5. The system according to claim 1, wherein: The fourth node is further used to determine a target expression for the resource consumption parameter based on the first variable and the second variable, when the number of data blocks to be copied in the M data blocks is a first variable and the number of data blocks to be erased is a second variable, wherein the value of the target expression represents the resource consumption parameter; and when the value of the target expression is minimum when the value of the first variable is the first number and the value of the second variable is the second number, the number of data blocks to be copied in the M data blocks is determined to be the first number, and the number of data blocks to be erased in the M data blocks is determined to be the second number.

6. A data processing method, characterized in that: include: When it is determined that the number of data blocks to be backed up is M data blocks, determining a first number of data blocks to be copied and a second number of data blocks to be erased among the M data blocks, where M is a positive integer greater than or equal to 2, and M is equal to the sum of the first number and the second number; among various combinations of the number of data blocks to be copied and the number of data blocks to be erased, the resource consumption parameter corresponding to the first number and the second number is minimized; performing the copy operation on the first number of data blocks among the M data blocks, and performing a first storage operation on the data blocks obtained by performing the copy operation; The erasure correction operation is performed on the second number of data blocks in the M data blocks, and a second storage operation is performed on the data blocks obtained by performing the erasure correction operation.

7. The method according to claim 6, characterized in that The determining that the number of data blocks to be copied among the M data blocks is a first number and the number of data blocks to be erased is a second number includes: The number of data blocks to be copied in the M data blocks is a first variable, and the number of data blocks to be erased is a first variable. When the quantity is the second variable, determining a target expression for the resource consumption parameter based on the first variable and the second variable, wherein the value of the target expression represents the resource consumption parameter; When the value of the target expression is minimum when the value of the first variable is the first number and the value of the second variable is the second number, the number of data blocks in the M data blocks on which the copy operation is to be performed is determined to be the first number, and the number of data blocks in the M data blocks on which the erasure operation is to be performed is determined to be the second number.

8. The method according to claim 7, characterized in that Before determining the number of data blocks in the M data blocks on which the copy operation is to be performed as the first number and determining the number of data blocks in the M data blocks on which the erasure operation is to be performed as the second number, the method further includes: When various value combinations of the number of data blocks to be copied and the number of data blocks to be erased include N value combinations, and each value combination in the N value combinations includes a value of the first variable and a value of the second variable, when the number of data blocks to be copied and the number of data blocks to be erased are each value combination in the N value combinations, determining a value of the target expression, thereby obtaining a total of N values of the target expression, where N is a positive integer greater than or equal to 2; Determine the minimum value among the N values, and determine the value of the first variable in a value combination corresponding to the minimum value as the first quantity, and determine the value of the second variable in the value combination as the second quantity, wherein the minimum value is the value of the target expression when the number of data blocks to be copied and the number of data blocks to be erased are the value combination.

9. The method according to claim 7, characterized in that Before determining the number of data blocks in the M data blocks on which the copy operation is to be performed as the first number and determining the number of data blocks in the M data blocks on which the erasure operation is to be performed as the second number, the method further includes: Within the value range of the first variable and the value range of the second variable, the target expression is solved for the minimum value to obtain the value of the first variable and the value of the second variable when the value of the target expression is the minimum value, and the value of the first variable when the value of the target expression is the minimum value is determined as the first quantity, and the value of the second variable when the value of the target expression is the minimum value is determined as the second quantity, wherein the minimum value is the value of the target expression when the value of the first variable is the first quantity and the value of the second variable is the second quantity.

10. The method according to claim 7, characterized in that Determining a target expression for the resource consumption parameter based on the first variable and the second variable includes: determining a first expression based on the first variable, and determining a second expression based on the first variable, wherein the first expression is used to represent the time consumed by performing the copy operation and the first storage operation on the data block to be copied, and the second expression is used to represent the storage space occupied by the data block to be copied; Determining a third expression based on the second variable and the third variable, and determining a fourth expression based on the third variable, wherein the third expression is used to represent the time consumed by performing the erasure operation and the second storage operation on the data blocks to be subjected to the erasure operation, the fourth expression is used to represent the storage space occupied by the data blocks to be subjected to the erasure operation, and the third variable is used to represent the number of data blocks obtained by performing the erasure operation on the data blocks equal to the second variable; The target expression is determined according to the first expression, the second expression, the third expression and the fourth expression.

11. The method according to claim 10, characterized in that Determining a first expression according to the first variable includes: The first expression is determined based on the first variable, the target size of the data block and the first transmission speed, wherein the target size of the data block is the average size of the M data blocks or the size of one of the M data blocks, the first transmission speed is the speed of transmitting data between the node where the data block obtained by performing the copy operation is located and the second node, and the first storage operation is used to store the data block obtained by performing the copy operation on the second node.

12. The method according to claim 10, characterized in that Determining a second expression according to the first variable includes: The second expression is determined according to the first variable and a target remaining number, wherein the target remaining number is used to represent an average value of a maximum number of data blocks allowed to be stored in a plurality of nodes.

13. The method according to claim 10, characterized in that Determining a third expression according to the second variable and the third variable includes: The third expression is determined based on the second variable, the third variable, the size of each data block in the data blocks obtained by performing the erasure operation on the data blocks having the second variable number of data blocks, the second transmission speed, and the target processing frequency, wherein the second transmission speed is the speed of transmitting data between the node where the data blocks obtained by performing the erasure operation are located and the third node, the second storage operation is used to store the data blocks obtained by performing the erasure operation on the third node, and the target processing frequency is used to represent the frequency at which the processor on the node where the data blocks obtained by performing the erasure operation are located processes data.

14. The method according to claim 10, characterized in that Determining a fourth expression according to the third variable includes: The fourth expression is determined according to the third variable and the target remaining number, wherein the target remaining number is used to represent an average value of a maximum number of data blocks allowed to be stored in a plurality of nodes.

15. The method according to claim 10, characterized in that Determining the target expression according to the first expression, the second expression, the third expression, and the fourth expression includes: The target expression is determined to be equal to an expression obtained by performing an addition operation on the first expression, the second expression, the third expression, and the fourth expression.

16. The method according to claim 6, characterized in that In the case where the M data blocks are in the first node, The performing the copy operation on the first number of data blocks in the M data blocks and performing the first storage operation on the data blocks obtained by performing the copy operation includes: performing the copy operation on the first number of data blocks in the M data blocks by the first node, and storing the data blocks obtained by performing the copy operation in the second node; The performing the erasure operation on the second number of data blocks in the M data blocks and performing a second storage operation on the data blocks obtained by performing the erasure operation includes: performing, by the first node, the erasure operation on the second number of data blocks in the M data blocks, and storing the data blocks obtained by performing the erasure operation in a third node; The second node is different from the first node, and the third node is different from the first node.

17. The method according to claim 6, characterized in that In the case where the M data blocks are in the first node, The performing the copy operation on the first number of data blocks in the M data blocks and performing the first storage operation on the data blocks obtained by performing the copy operation includes: performing the copy operation on the first number of data blocks in the M data blocks by the first node, and storing the data blocks obtained by performing the copy operation in the second node; The performing the erasure operation on the second number of data blocks in the M data blocks and performing a second storage operation on the data blocks obtained by performing the erasure operation includes: obtaining, by a fourth node, the second number of data blocks in the M data blocks, performing the erasure operation on the second number of data blocks in the M data blocks, and storing the data blocks obtained by performing the erasure operation in a third node; The second node is different from the first node, and the third node is different from the first node.

18. The method according to claim 6, characterized in that In the case where the M data blocks are in the first node, The performing the copy operation on the first number of data blocks among the M data blocks and performing the first storage operation on the data blocks obtained by performing the copy operation includes: obtaining the M data blocks by a fourth node, performing the copy operation on the first number of data blocks among the M data blocks, and storing the data blocks obtained by performing the copy operation in the second node; The performing the erasure operation on the second number of data blocks in the M data blocks and performing a second storage operation on the data blocks obtained by performing the erasure operation includes: performing the erasure operation on the second number of data blocks in the M data blocks by the fourth node, and storing the data blocks obtained by performing the erasure operation in the third node; The second node is different from the first node, and the third node is different from the first node.

19. The method according to any one of claims 6 to 15, characterized in that Before determining that the number of data blocks to be copied among the M data blocks is a first number and the number of data blocks to be erased is a second number, the method further includes: When the M data blocks are created in the memory space of the first node, the data blocks to be backed up are determined to be the M data blocks, wherein the M data blocks are created by the fourth node and the memory space of the first node is the remote memory space of the fourth node.

20. A computer-readable non-volatile storage medium, characterized in that: The computer non-volatile readable storage medium stores a computer program, wherein the computer program implements the method according to any one of claims 6 to 19 when executed by a processor.

Images