Data processing device and method and computer equipment

Abstract

The invention relates to a data processing device and method, computer equipment, a storage medium and a computer program product. The data processing device comprises: a data division module, which divides to-be-processed data into a plurality of data blocks; the computing storage nodes are used for acquiring at least one data block from the data division module and storing the data block; the plurality of erasure code check nodes are used for acquiring to-be-processed data from each calculation storage node, generating check code data based on the to-be-processed data and storing the check code data, and the back-end garrison process module is used for judging whether an abnormal calculation storage node exists or not; and when it is detected that the abnormal calculation storage node exists, obtaining to-be-processed data stored in the normal calculation storage node, generating recovery data based on the to-be-processed data and the check code data, and storing the recovery data in the normal calculation storage node. Through the setting, the validity of the data in the calculation storage node is guaranteed, the response abnormity probability of the processor is reduced, the consumed network bandwidth can be saved, and the utilization rate of the network is increased.

Description

Technical Field

[0001] The present application relates to the field of computer storage technology, and in particular to a data processing device, method, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of computer technology, the application of distributed storage systems has become more and more extensive. However, the hardware storage devices in distributed storage systems are prone to failures, such as host failures, hard disk failures, etc. Once a failure occurs, the data stored in it will inevitably be affected, which requires the distributed storage system to have a fault-tolerant mechanism.

[0003] A commonly used fault-tolerance method currently is to store data using erasure codes. However, if data is stored using erasure codes, data fault detection is generally performed only when a computing request is received, and the process of detecting data faults and repairing data needs to span multiple different hosts, which will cause some delays in the request response time of the storage system. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing device, method, computer equipment, storage medium and computer program product that can speed up the response speed of a distributed storage system in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing device, comprising:

[0006] A data partitioning module, used for partitioning the data to be processed into multiple data blocks;

[0007] A plurality of computing and storage nodes, used for acquiring at least one of the data blocks from the data partitioning module, and storing the acquired data blocks respectively;

[0008] A plurality of erasure code check nodes, used for obtaining the data to be processed from each of the computing and storage nodes, generating check code data based on the data to be processed, and storing the check code data respectively;

[0009] At least one of the computing and storage nodes or at least one of the erasure code check nodes is provided with a backend resident process module, and the backend resident process module is used to send periodic detection instructions to each of the computing and storage nodes, and determine whether there is an abnormal computing and storage node based on the detection result;

[0010] When the back-end resident process module detects the existence of an abnormal computing storage node, the erasure code check node is also used to obtain the pending data stored in the normal computing storage node, generate recovery data based on the pending data stored in the normal computing storage node and the check code data stored in the erasure code check node, and feed back the recovery data to be stored in the normal computing storage node.

[0011] In one embodiment, the data processing device further includes:

[0012] A client, the client is used to send a calculation instruction of the data to be processed to each of the calculation and storage nodes;

[0013] Each of the computing storage nodes is further used to perform parallel computing processing on the data to be processed stored in each node based on the computing instruction to be processed to obtain a computing result of the data to be processed; and transmit the computing result of the data to be processed back to the client;

[0014] The client also determines whether there is an abnormal computing storage node among the computing storage nodes based on the returned computing result of the data to be processed.

[0015] In one of the embodiments, the client is also used to send a recovery data calculation instruction to the computing storage node storing the recovery data; so that the computing storage node storing the recovery data performs calculation processing on the recovery data based on the recovery data calculation instruction, and after obtaining the recovery data calculation result, transmits the recovery data calculation result back to the client.

[0016] In one of the embodiments, the computing storage node also includes an acceleration card, and the backend daemon process module is disposed in the acceleration card.

[0017] In one of the embodiments, each of the computing and storage nodes also includes: a computing and storage module and a driver, wherein the driver is connected to the computing and storage module located in the same computing and storage node; the number of the back-end resident process module is one, and the back-end resident process module is located in the computing and storage node or in the erasure code check node, and is connected to the driver in each of the computing and storage nodes.

[0018] In one of the embodiments, each of the computing and storage nodes also includes: a computing and storage module and a driver; each of the computing and storage nodes is provided with the back-end resident process module, and the back-end resident process module is connected to the computing and storage module located in the same computing and storage node and the driver located in the same computing and storage node.

[0019] In a second aspect, the present application further provides a data processing method, which is applied to the data processing device described in any of the above embodiments, and the data processing method includes:

[0020] Divide the data to be processed into multiple data blocks and store them in multiple computing and storage nodes respectively;

[0021] The erasure code check node obtains the data to be processed stored by each of the computing and storage nodes, generates check code data based on the data to be processed, and stores the check code data to the multiple erasure code check nodes respectively; at least one of the computing and storage nodes or at least one of the erasure code check nodes has a backend resident process module;

[0022] The backend resident process module sends a periodic detection instruction to each of the computing and storage nodes, and determines whether there is an abnormal computing and storage node based on the detection result;

[0023] When the backend resident process module detects the presence of an abnormal computing storage node, the erasure code check node obtains the to-be-processed data stored in the normal computing storage node, and generates recovery data based on the to-be-processed data stored in the normal computing storage node and the check code data stored in the erasure code check node;

[0024] The erasure code check node feeds back the recovered data and stores it in a normal computing and storage node.

[0025] In one embodiment, the step of dividing the data to be processed into a plurality of data blocks and storing them in a plurality of computing and storage nodes respectively further includes:

[0026] The client sends a calculation instruction of the data to be processed to each of the computing and storage nodes;

[0027] Each of the computing storage nodes performs parallel computing processing on the data to be processed stored by each node based on the computing instruction to be processed to obtain a computing result of the data to be processed; and transmits the computing result of the data to be processed back to the client;

[0028] Based on the calculation result of the data to be processed transmitted back to the client, it is determined whether there is an abnormal computing storage node among the computing storage nodes.

[0029] In one embodiment, after the erasure code check node feeds back the recovered data and stores it in a normal computing storage node, the process further includes:

[0030] The client sends a recovery data calculation instruction to the calculation storage node storing the recovery data;

[0031] The computing storage node storing the recovery data performs computation processing on the recovery data based on the recovery data computation instruction to obtain a recovery data computation result.

[0032] In one of the embodiments, after the computing storage node storing the recovery data performs computing processing on the recovery data based on the recovery data computing instruction to obtain the recovery data computing result, the further comprises:

[0033] The computing storage node storing the recovery data transmits the recovery data calculation result back to the client.

[0034] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the data processing method described in any of the above embodiments is implemented.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the data processing method described in any of the above embodiments when executed by a processor.

[0036] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program product is executed by a processor, implements the data processing method described in any of the above embodiments.

[0037] The above-mentioned data processing device, method, computer equipment, storage medium and computer program product can automatically regenerate replacement data for abnormal data when an abnormal computing storage node is detected, thereby ensuring the response delay. At the same time, the recovery data is stored in the computing storage node, and the computing instructions can be responded to through other computing storage nodes, thereby saving consumed network bandwidth and increasing network utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of the structure of a data processing device in one embodiment;

[0039] Figure 2 is a schematic diagram of the structure of a data processing device in one embodiment;

[0040] Figure 3 is a schematic diagram of the structure of a data processing device in one embodiment;

[0041] Figure 4 A schematic diagram of the structure of a computing storage node in a data processing device in one embodiment;

[0042] Figure 5is a schematic diagram of the structure of a data processing device in one embodiment;

[0043] Figure 6 is a schematic diagram of the structure of a data processing device in one embodiment;

[0044] Figure 7 is a flow chart of a data processing method in one embodiment;

[0045] Figure 8 is a flow chart of a data processing method in one embodiment;

[0046] Fig. 9 is a flow chart of a data processing method in one embodiment;

[0047] Fig.10 is a flow chart of a data processing method in one embodiment;

[0048] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In one embodiment, Figure 1 As shown, a data processing device is provided, comprising:

[0051] A data partitioning module, at least partitioning the data to be processed into a plurality of data blocks;

[0052] The plurality of computing and storage nodes 102 obtain at least one data block from the data partitioning module and store the obtained data blocks respectively;

[0053] A plurality of erasure code check nodes 104 obtain the to-be-processed data from at least each computing storage node, generate check code data based on the to-be-processed data, and store the check code data respectively;

[0054] At least one computing storage node or at least one erasure code check node is provided with a backend resident process module 106, which at least sends a periodic detection instruction to each computing storage node, and determines whether there is an abnormal computing storage node based on the detection result;

[0055] When the backend resident process module detects the existence of an abnormal computing storage node, the erasure code check node is also used to obtain the pending data stored in the normal computing storage node, generate recovery data based on the pending data stored in the normal computing storage node and the check code data stored in the erasure code check node, and feed back the recovery data to be stored in the normal computing storage node.

[0056] The computing storage node 102 refers to a storage area that implements data redundancy through erasure coding, and is used to store data segmented by erasure coding.

[0057] Erasure coding is a data protection method that divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations.

[0058] In this embodiment, the data partitioning module can divide the data to be processed into multiple copies in the form of erasure coding (EC), so that each database contains a large block of continuous data to be processed, which is convenient for different computing and storage nodes 102 to perform distributed parallel computing. Under normal circumstances, there is almost no need for data interaction between data blocks of different computing and storage nodes 102, and the overall performance of the system is greatly improved.

[0059] The check code data may include the data to be processed in the data blocks of each computing storage node 102 , so that when an abnormality occurs in the data blocks of any computing storage node 102 , the erasure code check node 104 can perform data recovery based on the check code data in the erasure code check node 104 .

[0060] A node refers to a computer or other device connected to a network with an independent address and the ability to transmit or receive data. The computing storage node 102 and the erasure code check node 104 can be workstations, clients, network users or personal computers, and can also be servers, printers and other network-connected devices. Every workstation, server, terminal device, network device, that is, a device with its own unique network address can be a computing storage node 102 or an erasure code check node 104.

[0061] The computing storage node 102 and the erasure code check node 104 are used to distinguish between the device where the data to be processed is located and the device where the check code data is located, and each computing storage node 102 can store at least one data block for storing the data to be processed, and each erasure code check node 104 can store at least one data block for storing the check code data. Specifically, the data partitioning module 102 divides the data to be processed into n parts in the form of erasure codes, and stores them in n data blocks respectively. The computing storage node 102 stores the n data blocks in m computing storage nodes respectively, where m≤n. The following takes m=n as an example. Then, the erasure code check node 104 generates at least one check code data according to the data to be processed contained in the n computing storage nodes. The check code data can be expressed as i∈k where represents XOR processing, a, b, c, d...n are all non-zero natural numbers, and their values ​​can be preset by the staff, k represents the type of check code data, a i 、b i 、c i d i ... i They respectively represent the non-zero natural numbers corresponding to the i-th check code data.

[0062] Among them, the value of n affects the cost of data recovery. The smaller the value of n, the smaller the dispersion of the data to be processed, the greater the impact of data anomalies, and the greater the reconstruction cost; the larger the value of n, the greater the network and disk load increased by multiple data copies, and the greater the reconstruction cost. The value of k affects reliability and storage cost. The larger the value of k, the higher the tolerance for data anomalies; the smaller the value of k, the lower the data redundancy. Staff can pre-set the values ​​of n and k according to actual needs.

[0063] As an example, when the data partitioning module divides the to-be-processed data into four parts in the form of erasure code, four computing and storage nodes can respectively store four data blocks corresponding to the four to-be-processed data. The four computing and storage nodes are named A, B, C, and D respectively. The erasure code check node obtains the to-be-processed data in the four data blocks from the computing and storage nodes A, B, C, and D, and generates two types of check code data based on the to-be-processed data in the four data blocks. The first type of check code data corresponds to a 1 、b 1 、c 1 d 1 They are 1, 1, 1, 1 respectively. The a corresponding to the second check code data 2 、b 2 、c 2 d 2 The generated verification code data are 1, 2, 3, and 4 respectively. and The data can be stored in two erasure code check nodes respectively, which are named E and F respectively.

[0064] The back-end daemon process module 106 can be set in at least one computing storage node 102 and at least one erasure code check node 104. It should be noted that any computing storage node 102 is connected to the back-end daemon process module 106. It can be understood that when there is no back-end daemon process module 106 in the computing storage node 102, the computing storage node 102 can be connected to any back-end daemon process module 106 in other computing storage nodes 102 or in the erasure code check node 104.

[0065] The back-end resident process module 106 is used to send periodic detection instructions to the computing storage node 102 to perform periodic detection on the computing storage node 102 and obtain corresponding detection results. The back-end resident process module 106 can determine whether there is an abnormal computing storage node based on the detection results. Among them, the periodic detection instruction can be, for example, reading the data to be processed stored in the computing storage node 102. When the data to be processed stored in any data block in the computing storage node 102 is lost, missing, or cannot be read normally, the back-end resident process module 106 can determine that the current computing storage node is abnormal based on the detection results.

[0066] Recovery data refers to data used to replace the data to be processed in the computing storage node where the exception occurs.

[0067] In this embodiment, if the backend daemon process module 106 detects that the data to be processed in the computing storage node 102 is abnormal, the computing storage node 102 with the abnormality will not be able to respond normally, which will cause the response of the computing storage node 102 with the abnormality to stagnate, thereby affecting the user experience. The erasure code check node 104 in this embodiment can quickly generate recovery data to replace the data to be processed in the abnormal computing storage node 102 based on the data to be processed in the remaining computing storage nodes 102 that have not been abnormal and the check code data in the erasure code check node 104 when the abnormal computing storage node 102 appears. As an example, the abnormal computing storage node 102 may refer to the computing storage node 102 where the data to be processed in any data block is lost, missing, or cannot be read normally.

[0068] After the backend resident process module 106 detects an abnormal computing storage node 102, the erasure code check node 104 can extract the data to be processed from the computing storage node 102 that has not detected the abnormality, and process it to obtain the recovery data based on the check code data stored in itself, and then send the recovery data to any normal computing storage node 102 for storage.

[0069] As an example, the data partitioning module divides the data to be processed into four data blocks, which are stored in four computing and storage nodes respectively. The four computing and storage nodes are named A, B, C, and D. When the back-end resident process module 106 detects that the computing and storage node A is abnormal, the erasure code check node generates recovery data based on the data to be processed contained in the computing and storage nodes B, C, and D and the check code data stored in itself to replace the data to be processed contained in the computing and storage node A. The erasure code check node then sends the recovery data to any one of the computing and storage nodes B, C, and D for storage.

[0070] In this embodiment, the erasure code check node 104 stores the recovery data in any normal computing storage node 102. Through this setting, the recovery data can share a node with the pending data in the remaining computing storage nodes 102 for data writing and reading, saving node interaction loss and reducing power consumption.

[0071] like Figure 2 As shown, in some optional embodiments, the data processing device further includes:

[0072] The client 108 sends at least a calculation instruction of the data to be processed to each computing and storage node 102;

[0073] Each computing storage node 102 is also used to perform parallel computing on the data to be processed stored by each node based on the computing instruction to be processed to obtain the computing result of the data to be processed; and transmit the computing result of the data to be processed back to the client 108;

[0074] The client 108 also determines whether there is an abnormal computing storage node in each computing storage node 102 based on the returned computing results of the data to be processed.

[0075] The data to be processed calculation instruction refers to an instruction issued for the data to be processed contained in each computing storage node 102 to read or perform related calculation processing on the data to be processed. As an example, the data blocks in each computing storage node 102 respectively carry an identity tag, and the identity tag can be composed of at least one of letters, characters or numbers. The identity tag is used to uniquely identify the data block, and the data to be processed calculation instruction can carry at least one identity tag to identify the data block corresponding to the data to be processed calculation instruction.

[0076] Correspondingly, the calculation processing result refers to the data to be processed read from each calculation storage node 102 or the result after the relevant calculation processing is performed on the data to be processed in each calculation storage node 102.

[0077] In this embodiment, the client 108 can match the corresponding data block according to the identity tag carried by the computing instruction of the data to be processed, and determine the computing storage node 102 where the matched database is located in one step, and then issue the computing instruction of the data to be processed to the matched computing storage node 102. The computing storage node 102 reads the data to be processed in the corresponding data block or performs related computing processing according to the computing instruction of the data to be processed. After obtaining the computing processing result, the computing storage node 102 returns the computing processing result to the client 108.

[0078] When the data to be processed stored in the computing storage node 102 is lost, missing, or cannot be read normally, the client 108 can determine whether there is an abnormal computing storage node 102 based on the computing processing result.

[0079] In some optional embodiments, the client 108 is also used to send a recovery data calculation instruction to the computing storage node storing the recovery data; so that the computing storage node storing the recovery data performs calculation processing on the recovery data based on the recovery data calculation instruction, and after obtaining the recovery data calculation result, transmits the recovery data calculation result back to the client.

[0080] like Figure 3 As shown, as an example, after the client 108 determines that there is an abnormal computing storage node 102, the client 108 can generate a recovery instruction carrying the identity tag of the data block where the abnormality actually occurs in the abnormal computing storage node 102, and send the recovery instruction to the erasure code check node 104. The erasure code check node 104 obtains the to-be-processed data stored in the normal computing storage node 102 according to the recovery instruction, and generates recovery data based on the normal to-be-processed data and the check code data, and stores the recovery data in any data block in the normal computing storage node. Alternatively, the erasure code check node 104 can generate a new data block in the normal computing storage node for storing the recovery data.

[0081] As an example, when generating recovery data, the erasure code check node 104 of this embodiment can assign an identity tag corresponding to the data block where the abnormality actually occurred in the abnormal computing storage node to the recovery data, and remove the identity tag carried by the to-be-processed data originally stored in the data block where the abnormality actually occurred in the abnormal computing storage node.

[0082] Subsequently, after the client 108 issues a recovery instruction, the client 108 can automatically generate a recovery data calculation instruction based on the identity tag of the abnormal computing storage node 102. The client 108 can match the data block where the recovery data is located based on the identity tag carried by the recovery data calculation instruction, and determine the computing storage node 102 corresponding to the data block where the recovery data is located in one step, and then send the recovery data calculation instruction to the matched computing storage node 102.

[0083] Furthermore, the computing storage node 102 storing the recovery data performs corresponding reading processing or calculation processing on the recovery data according to the recovery data calculation instruction, and after obtaining the recovery data calculation result, the computing storage node 102 returns the recovery data calculation result to the client 108.

[0084] like Figure 4 As shown, in some optional embodiments, the computing storage node 102 also includes an acceleration card 110, and the backend daemon process module 106 is disposed in the acceleration card 110.

[0085] The accelerator card 110 can unload the workload of the computing storage node 102 and release the resources of the computing storage node 102 .

[0086] like Figure 5 As shown, in some optional embodiments, each computing storage node 102 also includes: a computing storage module 1022a and a driver 1024a, and the driver 1024a is connected to the computing storage module 1022a located in the same computing storage node 102; the number of back-end resident process modules 106 is one, and the back-end resident process module 106 is located in the computing storage node 102 or in the erasure code check node 104, and is connected to the driver 1024a in each computing storage node 102.

[0087] As an example, the erasure code check node 104 may also include a computing storage module 1042a and a driver 1044a, wherein the driver 1044a is connected to the computing storage module 1042a located in the same erasure code check node 104; when the only back-end daemon process module 106 is located in a computing storage node 102, the back-end daemon process module 106 can be connected to the drivers 1024a of all other computing storage nodes 102 and the drivers 1044a of the erasure code check nodes 104.

[0088] The backend daemon process module 106 can send periodic detection instructions to the driver 1024a of each computing storage node 102. Each computing storage node 102 sends the detection results to the backend daemon process module 106 through the driver 1024a. The backend daemon process module 106 then determines whether there are abnormal computing storage nodes 102 based on the detection results.

[0089] like Figure 6 As shown, in some optional embodiments, each computing storage node 102 also includes: a computing storage module 1022b and a driver 1024b; each computing storage node 102 is provided with a back-end resident process module 106, and the back-end resident process module 106 is connected to the computing storage module 1022b located in the same computing storage node 102 and the driver 1024b located in the same computing storage node.

[0090] As an example, the erasure code check node 104 may also include a computing storage module 1042b and a driver 1044b, and the driver 1044b is connected to the computing storage module 1042b located in the same erasure code check node 104; when each computing storage node 102 is provided with a back-end resident process module 106 and each erasure code check node 104 is provided with a back-end resident process module 106, the back-end resident process module 106 in the computing storage node 102 can be respectively connected to the computing storage module 1022b and the driver 1024b located in the same computing storage node 102, and the back-end resident process module 106 in the erasure code check node 104 can be respectively connected to the computing storage module 1042b and the driver 1044b located in the same erasure code check node 104.

[0091] The backend daemon process module 106 can send periodic detection instructions to the driver 1024a of each computing storage node 102. Each computing storage node 102 sends the detection results to the backend daemon process module 106 through the driver 1024a. The backend daemon process module 106 then determines whether there are abnormal computing storage nodes 102 based on the detection results.

[0092] In one embodiment, a data processing method is provided. It should be noted that the data processing method is applied to the data processing device described in any of the above embodiments. Figure 7 As shown, the data processing method includes:

[0093] Step 202: Divide the data to be processed into multiple data blocks, and store them in multiple computing and storage nodes respectively.

[0094] Step 204: The erasure code check node obtains the data to be processed stored in each computing storage node, and generates check code data based on the data to be processed, and stores them in multiple erasure code check nodes respectively; at least one computing storage node or at least one erasure code check node has a backend resident process module.

[0095] Step 206: The backend resident process module sends periodic detection instructions to each computing and storage node, and determines whether there are abnormal computing and storage nodes based on the detection results.

[0096] Step 208: When the backend resident process module detects the existence of an abnormal computing storage node, the erasure code check node obtains the pending data stored in the normal computing storage node, and generates recovery data based on the pending data stored in the normal computing storage node and the check code data stored in the erasure code check node.

[0097] Step 210: The erasure code check node feeds back the recovered data and stores it in a normal computing and storage node.

[0098] The above data processing method can split the data in the form of erasure codes, and store the split data to be processed in the computing and storage nodes respectively. The erasure code verification node can obtain the data to be processed in the computing and storage node, generate verification code data and store it. The back-end resident process module can perform periodic anomaly detection on each computing and storage node, thereby realizing automatic anomaly detection of data in the erasure code data block, ensuring the validity of the data to be processed in the computing and storage node. When there is an abnormal computing and storage node, it can generate recovery data based on the data to be processed contained in the remaining computing and storage nodes that have not detected the abnormality, and store the recovery data in any normal computing and storage node. In this way, the automatic regeneration of abnormal data is realized, and the fault tolerance rate of abnormal data is improved.

[0099] like Figure 8 As shown, in some optional embodiments, after step 202, the method further includes:

[0100] Step 2032: The client sends a calculation instruction for the data to be processed to each computing and storage node;

[0101] Step 2034: Each computing storage node performs parallel computing on the data to be processed stored by each node based on the computing instruction to be processed to obtain the computing result of the data to be processed; and transmits the computing result of the data to be processed back to the client;

[0102] Step 2036: Based on the calculation results of the data to be processed transmitted back to the client, determine whether there are any abnormal computing and storage nodes among the computing and storage nodes.

[0103] like Fig. 9 As shown, in some optional embodiments, after step 202, the method further includes:

[0104] Step 2038: The client sends a recovery data calculation instruction to the calculation storage node storing the recovery data;

[0105] Step 20310: The computing storage node storing the recovery data performs computation on the recovery data based on the recovery data computation instruction to obtain a recovery data computation result.

[0106] like Fig.10As shown, in some optional embodiments, after step 20310, the method further includes:

[0107] Step 20312: The computing storage node storing the recovery data transmits the calculation result of the recovery data back to the client.

[0108] As an example, when computing storage node A is abnormal, the processor generates the corresponding recovery data of computing storage node A and randomly saves it to the normal computing storage node B. When the client Fuan issues a recovery data calculation instruction carrying the identity tag of the database in computing storage node A, computing storage node B performs calculation processing on the recovery data according to the recovery data calculation instruction, obtains the recovery data calculation result of the recovery data, and then returns the recovery data calculation result to the client through computing storage node B. Compared with the method in which computing storage node A and computing storage node B return the calculation results to the client separately, the number of participating nodes is saved. By this method, the waste of computing resources can be reduced.

[0109] In some optional embodiments, the data processing method further includes: when at least two abnormal data blocks are detected, generating recovery data corresponding to the abnormal data blocks in sequence according to the abnormal data blocks and a preset node processing order.

[0110] As an example, each data block is pre-set with a corresponding node label, and the node label can be composed of at least one of letters, characters or numbers. The node label is used to uniquely identify the computing storage node, and the processor pre-stores a sequence table of node labels to indicate the node processing order of different computing storage nodes. The sequence table can be sorted by the staff according to the importance of different computing storage nodes, the importance of the data to be processed stored in different computing storage nodes, or it can be determined according to the number of data blocks in different computing storage nodes. As an example, the importance of the computing storage node can be distinguished according to the number of connected computing storage nodes, and the importance of the data can be distinguished according to the probability of the data being used.

[0111] As mentioned above, the processor can tolerate the failure of k data blocks. When the processor detects that at least two data blocks are abnormal, it first determines whether the abnormal data blocks are in the same computing and storage node. If not, the corresponding processing order is matched from the sequence table pre-saved by the processor according to the node labels of the computing and storage nodes where the abnormal data blocks are located, and then the corresponding recovery data is generated in sequence according to the processing order of the node labels. If they are in the same computing and storage node, it can be understood that multiple data blocks are abnormal in the same computing and storage node. At this time, the data to be processed originally stored in the data blocks with abnormalities in the computing and storage node can be sorted according to the amount of data, and the recovery data corresponding to the data blocks with larger data volume is generated first.

[0112] In this embodiment, according to the pre-set node processing order, it is possible to preferentially generate recovery data corresponding to the data to be processed stored in more important computing storage nodes, or preferentially generate recovery data corresponding to the data to be processed in the data blocks with larger data volume in the same computing storage node. Through this setting, the importance of data is layered, the failure probability of the processor during response is reduced, and the validity of the data stored in the processor is guaranteed.

[0113] In the above data processing method, anomaly detection of each computing and storage node is performed through the back-end resident process, which realizes automatic anomaly detection of the data to be processed in the computing and storage nodes, ensures the validity of the data to be processed in the computing and storage nodes, and when an abnormal computing and storage node is detected, the automatic regeneration of the abnormal data to be processed can be realized, thereby improving the fault tolerance rate of the abnormal data to be processed. In addition, the data processing method can store the recovery data in other normal computing and storage nodes, and respond to the computing instructions of the data to be processed through other computing and storage nodes, which can save the consumed network bandwidth and increase the utilization rate of the network.

[0114] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0116] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0118] Divide the data to be processed into multiple data blocks and store them in multiple computing and storage nodes respectively;

[0119] The erasure code check node obtains the to-be-processed data stored in each computing and storage node, and generates check code data based on the to-be-processed data, and stores the data in multiple erasure code check nodes respectively; at least one computing and storage node or at least one erasure code check node has a backend resident process module;

[0120] The backend resident process module sends periodic detection instructions to each computing and storage node, and determines whether there are abnormal computing and storage nodes based on the detection results;

[0121] When the backend resident process module detects the presence of an abnormal computing storage node, the erasure code check node obtains the to-be-processed data stored in the normal computing storage node, and generates recovery data based on the to-be-processed data stored in the normal computing storage node and the check code data stored in the erasure code check node;

[0122] The erasure code check node will recover the data and store it in the normal computing and storage nodes.

[0123] In some optional embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0124] The client sends the data calculation instructions to be processed to each computing and storage node;

[0125] Each computing storage node performs parallel computing on the data to be processed stored by each node based on the computing instructions to be processed to obtain the computing results of the data to be processed; and transmits the computing results of the data to be processed back to the client;

[0126] Based on the calculation results of the data to be processed returned to the client, it is determined whether there are any abnormal computing and storage nodes among the computing and storage nodes.

[0127] In some optional embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0128] The client sends a recovery data calculation instruction to the calculation storage node storing the recovery data;

[0129] The computing storage node storing the recovery data performs computation processing on the recovery data based on the recovery data computation instruction to obtain a recovery data computation result.

[0130] In some optional embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0131] The computing storage node storing the recovery data transmits the calculation result of the recovery data back to the client.

[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor:

[0133] Divide the data to be processed into multiple data blocks and store them in multiple computing and storage nodes respectively;

[0134] The erasure code check node obtains the to-be-processed data stored in each computing and storage node, and generates check code data based on the to-be-processed data, and stores the data in multiple erasure code check nodes respectively; at least one computing and storage node or at least one erasure code check node has a backend resident process module;

[0135] The backend resident process module sends periodic detection instructions to each computing and storage node, and determines whether there are abnormal computing and storage nodes based on the detection results;

[0136] When the backend resident process module detects the presence of an abnormal computing storage node, the erasure code check node obtains the to-be-processed data stored in the normal computing storage node, and generates recovery data based on the to-be-processed data stored in the normal computing storage node and the check code data stored in the erasure code check node;

[0137] The erasure code check node will recover the data and store it in the normal computing and storage nodes.

[0138] In some optional embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0139] The client sends the data calculation instructions to be processed to each computing and storage node;

[0140] Each computing storage node performs parallel computing on the data to be processed stored by each node based on the computing instructions to be processed to obtain the computing results of the data to be processed; and transmits the computing results of the data to be processed back to the client;

[0141] Based on the calculation results of the data to be processed returned to the client, it is determined whether there are any abnormal computing and storage nodes among the computing and storage nodes.

[0142] In some optional embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0143] The client sends a recovery data calculation instruction to the calculation storage node storing the recovery data;

[0144] The computing storage node storing the recovery data performs computation processing on the recovery data based on the recovery data computation instruction to obtain a recovery data computation result.

[0145] In some optional embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0146] The computing storage node storing the recovery data transmits the calculation result of the recovery data back to the client.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing device, characterized in that: include: A data partitioning module, at least partitioning the data to be processed into a plurality of data blocks; A plurality of computing and storage nodes obtain at least one of the data blocks from the data partitioning module and store the obtained data blocks respectively; A plurality of erasure code check nodes, at least obtaining the data to be processed from each of the computing and storage nodes, generating check code data based on the data to be processed, and storing the check code data respectively; At least one of the computing and storage nodes or at least one of the erasure code check nodes is provided with a back-end resident process module, and the back-end resident process module at least sends a periodic detection instruction to each of the computing and storage nodes, and determines whether there is an abnormal computing and storage node based on the detection result; When the back-end resident process module detects the existence of an abnormal computing storage node, the erasure code check node is also used to obtain the pending data stored in the normal computing storage node, generate recovery data based on the pending data stored in the normal computing storage node and the check code data stored in the erasure code check node, and feed back the recovery data to be stored in the normal computing storage node.

2. The data processing device according to claim 1, characterized in that: The data processing device further comprises: A client, the client at least sending a calculation instruction of the data to be processed to each of the calculation and storage nodes; Each of the computing storage nodes is further used to perform parallel computing processing on the data to be processed stored in each node based on the computing instruction to be processed to obtain a computing result of the data to be processed; and transmit the computing result of the data to be processed back to the client; The client also determines whether there is an abnormal computing storage node among the computing storage nodes based on the returned computing result of the data to be processed.

3. The data processing device according to claim 2, characterized in that: The client is also used to send a recovery data calculation instruction to the calculation storage node storing the recovery data; so that the calculation storage node storing the recovery data performs calculation processing on the recovery data based on the recovery data calculation instruction, and after obtaining the recovery data calculation result, transmits the recovery data calculation result back to the client.

4. The data processing device according to claim 1, characterized in that: The computing storage node also includes an acceleration card, and the backend daemon process module is arranged in the acceleration card.

5. The data processing device according to any one of claims 1 to 4, characterized in that: Each of the computing and storage nodes also includes: a computing and storage module and a driver, and the driver is connected to the computing and storage module located in the same computing and storage node; the number of the back-end resident process module is one, and the back-end resident process module is located in the computing and storage node or in the erasure code check node, and is connected to the driver in each of the computing and storage nodes.

6. The data processing device according to any one of claims 1 to 4, characterized in that: Each of the computing and storage nodes also includes: a computing and storage module and a driver; each of the computing and storage nodes is provided with the back-end resident process module, and the back-end resident process module is connected to the computing and storage module located in the same computing and storage node and the driver located in the same computing and storage node.

7. A data processing method, characterized in that: Applied to the data processing device according to any one of claims 1 to 6, the data processing method comprises: Divide the data to be processed into multiple data blocks and store them in multiple computing and storage nodes respectively; The erasure code check node obtains the data to be processed stored by each of the computing and storage nodes, generates check code data based on the data to be processed, and stores the check code data to the multiple erasure code check nodes respectively; at least one of the computing and storage nodes or at least one of the erasure code check nodes has a backend resident process module; The backend resident process module sends a periodic detection instruction to each of the computing and storage nodes, and determines whether there is an abnormal computing and storage node based on the detection result; When the backend resident process module detects the presence of an abnormal computing storage node, the erasure code check node obtains the to-be-processed data stored in the normal computing storage node, and generates recovery data based on the to-be-processed data stored in the normal computing storage node and the check code data stored in the erasure code check node; The erasure code check node feeds back the recovered data and stores it in a normal computing and storage node.

8. The method according to claim 7, characterized in that After dividing the data to be processed into a plurality of data blocks and storing them in a plurality of computing and storage nodes respectively, the method further includes: The client sends a calculation instruction of the data to be processed to each of the computing and storage nodes; Each of the computing storage nodes performs parallel computing processing on the data to be processed stored by each node based on the computing instruction to be processed to obtain a computing result of the data to be processed; and transmits the computing result of the data to be processed back to the client; Based on the calculation result of the data to be processed transmitted back to the client, it is determined whether there is an abnormal computing storage node among the computing storage nodes.

9. The method according to claim 7, characterized in that: After the erasure code check node feeds back the recovered data and stores it in a normal computing and storage node, the method further includes: The client sends a recovery data calculation instruction to the calculation storage node storing the recovery data; The computing storage node storing the recovery data performs computation processing on the recovery data based on the recovery data computation instruction to obtain a recovery data computation result.

10. The method according to claim 9, characterized in that After the computing storage node storing the recovery data performs computing processing on the recovery data based on the recovery data computing instruction to obtain the recovery data computing result, the method further includes: The computing storage node storing the recovery data transmits the recovery data calculation result back to the client.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data processing method according to any one of claims 7 to 10 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 7 to 10 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 7 to 10 are implemented.

Images