Data processing method and device, equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, equipment, a readable storage medium and a program product. The method comprises the steps of obtaining a chance table corresponding to a storage array, performing reading operation on each storage address to obtain a first reading result, performing routing inspection processing on each to-be-processed data according to the first reading result to obtain a plurality of pieces of first routing inspection data, and performing routing inspection processing on each piece of to-be-processed data according to each piece of identification data and each piece of first routing inspection data to obtain a plurality of pieces of second routing inspection data. The opportunity table comprises multiple storage addresses and identification data of the to-be-processed data stored in the storage addresses, and the multiple storage addresses are storage addresses in at least one storage node in the storage array. By adopting the method, the access efficiency of accessing each storage node by the host can be improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method, device, equipment, readable storage medium and program product. Background Art

[0002] With the development of storage technology, flash memory (NAND FLASH) is widely used as a storage medium. However, flash memory has the characteristic that the data stored inside will be lost over time. Therefore, in a storage array composed of multiple flash memories, the flash memory is used as a storage node, and the control node will regularly and actively read the storage data in each storage node for consistency inspection, so as to repair the erroneous data; in addition, as the amount of stored data increases, duplicate data may be stored in each storage node. In order to increase the capacity of the available space in the flash memory, the control node will also regularly and actively read the storage data in each storage node and clean up the duplicate storage data.

[0003] However, the above data processing methods will affect the access efficiency of the host to each storage node. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device, equipment, readable storage medium and program product that can improve access efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Obtaining an opportunity table corresponding to the storage array, the opportunity table including a plurality of storage addresses and identification data of to-be-processed data stored in each storage address, the plurality of storage addresses being storage addresses in at least one storage node in the storage array;

[0007] Performing a read operation on each storage address to obtain a first read result;

[0008] According to the first reading result, each data to be processed is inspected to obtain a plurality of first inspection data, and according to each identification data and each first inspection data, duplicate data is deleted from each storage address.

[0009] In the above embodiment, the control node only needs to actively read the data once according to the storage address stored in the opportunity table to complete the inspection processing and duplicate data deletion processing of the data to be processed in each storage address, thereby avoiding the problem of excessive bandwidth occupation of the control node and each storage node in the traditional technology. The technical solution provided in the present application occupies less bandwidth between the control node and each storage node, provides more bandwidth for the host to access the storage node through the control node, and improves the access efficiency of the host to each storage node.

[0010] In one embodiment, according to the first reading result, each data to be processed is inspected to obtain a plurality of first inspection data, including:

[0011] For a storage address, if the first read result includes the to-be-processed data corresponding to the storage address, verifying the to-be-processed data according to the first verification code included in the first read result;

[0012] The first inspection data is obtained according to the verification result.

[0013] In the above embodiment, after the data to be processed is successfully read, the data to be processed can be verified, and correct first inspection data can be obtained according to the verification result, thereby improving the correctness of the data.

[0014] In one embodiment, obtaining first inspection data according to the verification result includes:

[0015] If the verification result is a verification failure, metadata corresponding to the storage address is obtained, and data repair is performed on the data to be processed according to the metadata to obtain first inspection data;

[0016] If the verification result is successful, it is determined that the data to be processed is the first inspection data.

[0017] In the above embodiment, when the verification result is a verification failure, the data to be processed is repaired according to the metadata. When the verification result is a verification success, the data to be processed is directly used as the first inspection data. There is no need to repair all the data to be processed, thereby reducing unnecessary waste of computing resources.

[0018] In one embodiment, the method further comprises:

[0019] If the first reading result does not include the to-be-processed data corresponding to the storage address, obtaining metadata corresponding to the storage address;

[0020] Data repair is performed on the data to be processed based on the metadata to obtain first inspection data.

[0021] In the above embodiment, when the data to be processed cannot be read from the storage address, the control node performs inspection processing on the data to be processed, repairs the data that cannot be read due to data errors, and improves the correctness of the data.

[0022] In one embodiment, deduplication processing is performed on each storage address according to each identification data and each first inspection data, including:

[0023] Determine multiple target storage addresses from the opportunity table, the multiple target storage addresses corresponding to the same identification data;

[0024] Comparing target first inspection data corresponding to a plurality of target storage addresses;

[0025] If the first inspection data of each target is the same, deduplication processing is performed on the target to-be-processed data stored in each target storage address.

[0026] In the above embodiment, after each data to be processed is inspected, deduplication processing is continued on the data in each storage address, thereby avoiding the problem of excessive bandwidth occupation caused by multiple active readings of the storage node due to subsequent deduplication processing by the control node, broadening the available bandwidth for the host to access the storage node, and improving the access efficiency of the host to the storage node.

[0027] In one embodiment, deduplication processing is performed on target data to be processed stored in each target storage address, including:

[0028] Obtain the static time corresponding to each target data to be processed;

[0029] Determine a reserved storage address corresponding to the shortest static time among the static times;

[0030] The target data to be processed in the target storage addresses other than the reserved storage addresses in each target storage address are deleted.

[0031] In the above embodiment, the target data to be processed in the reserved storage address corresponding to the shortest static time is retained, and the target data to be processed in other target storage addresses are deleted, so that the target data to be processed finally retained is the latest updated data, ensuring the timeliness of the data.

[0032] In one embodiment, the method further comprises:

[0033] De-duplicate the first inspection data of each target.

[0034] In the above embodiment, duplicate data are deleted from the same target first inspection data, thereby ensuring that duplicate target first inspection data will not be rewritten into the target storage address from which the data has been deleted.

[0035] In one embodiment, the method further comprises:

[0036] For each first inspection data, if the first inspection data is data-repaired data, the first inspection data is sent to the corresponding storage node with a first update write instruction, and the first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first inspection data according to the first inspection data.

[0037] In the above embodiment, the first inspection data after data repair is used to update the data to be processed in the corresponding storage address, thereby ensuring the correctness of the data in the storage address.

[0038] In one embodiment, the method further comprises:

[0039] Add a data processing completion mark to each storage address.

[0040] In the above embodiment, adding a data processing completion mark to each storage address can avoid repeated data processing of the storage address, thereby reducing unnecessary waste of resources.

[0041] In one embodiment, obtaining an opportunity table corresponding to a storage array includes:

[0042] Acquire multiple write instructions sent by the host, where the write instructions include write data and write addresses corresponding to the write data;

[0043] If the occupancy rate of the cache in the control node is greater than the preset occupancy rate threshold, each write data is written to each write address, and a write success prompt message is returned to the host;

[0044] When the occupancy rate is less than a preset occupancy rate threshold, identification data corresponding to each write data is obtained according to each write address, and an opportunity table is obtained according to each identification data.

[0045] In the above embodiment, when the cache pressure is too great, the write data is directly written to the address, and a write success prompt message is returned to the host to prompt the host to continue writing data, and then wait for the appropriate actual calculation of the identification data of each write data, which greatly alleviates the calculation pressure of the cache and improves the calculation efficiency of the identification data.

[0046] In one embodiment, obtaining identification data corresponding to each written data according to each written address includes:

[0047] Performing a read operation on each write address to obtain a second read result;

[0048] According to the second reading result, each written data is inspected to obtain a plurality of second inspection data;

[0049] Each second inspection data is processed according to a preset condition to obtain identification data corresponding to each write data.

[0050] In the above embodiment, each written data is firstly inspected, and then the identification data corresponding to each written data is calculated using the second inspection data obtained by the inspection, thereby ensuring the correctness of the identification data.

[0051] In one embodiment, according to the second reading result, each written data is inspected to obtain a plurality of second inspection data, including:

[0052] For a write address, if the second read result does not include the write data corresponding to the write address, then data repair is performed on the write data according to the metadata corresponding to the write address to obtain second inspection data;

[0053] If the second read result includes write data corresponding to the write address, the data to be processed is verified according to the second verification code included in the second read result. If the verification result is a verification failure, the write data is repaired according to the metadata corresponding to the write address to obtain second inspection data. If the verification result is a verification success, it is determined that the write data is the second inspection data.

[0054] In the above embodiment, by performing patrol processing on each written data, the correctness of the identification data and the correctness of the data in each written address can be ensured.

[0055] In one embodiment, the method further comprises:

[0056] For each second inspection data, if the second inspection data is data repaired, the second inspection data is carried in a second update write instruction and sent to the corresponding storage node, and the second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0057] In the above embodiment, the data written in the write address is updated by using the second inspection data that has been inspected, thereby improving the correctness of the data in the write address.

[0058] In a second aspect, the present application further provides a data processing device, comprising:

[0059] An acquisition module, used to acquire an opportunity table corresponding to the storage array, wherein the opportunity table includes a plurality of storage addresses and identification data of the to-be-processed data stored in each storage address, wherein the plurality of storage addresses are storage addresses in at least one storage node in the storage array;

[0060] A reading module, used for performing a reading operation on each storage address to obtain a first reading result;

[0061] The processing module is used to perform inspection processing on each to-be-processed data according to the first reading result to obtain a plurality of first inspection data, and perform duplicate data deletion processing on each storage address according to each identification data and each first inspection data.

[0062] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method of the first aspect described above are implemented.

[0064] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in the first aspect are implemented.

[0065] The above-mentioned data processing method, device, equipment, readable storage medium and program product can obtain a first reading result by acquiring the opportunity table corresponding to the storage array and performing a read operation on each storage address. According to the first reading result, each data to be processed can be inspected to obtain multiple first inspection data, and duplicate data can be deleted from each storage address according to each identification data and each first inspection data, wherein the opportunity table includes multiple storage addresses and identification data of the data to be processed stored in each storage address, and the multiple storage addresses are storage addresses in at least one storage node in the storage array. In this way, the control node only needs to perform active data reading once according to the storage address stored in the opportunity table to complete the inspection and duplicate data deletion of the data to be processed in each storage address, avoiding the problem of excessive bandwidth occupation of the control node and each storage node in the traditional technology that the data in the storage node needs to be repeatedly read to complete the inspection and duplicate data deletion of the data. The technical solution provided by the present application occupies less bandwidth between the control node and each storage node, provides more bandwidth for the host to access the storage node through the control node, and improves the access efficiency of the host to each storage node. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 is a graph showing the relationship between storage data and static time in the flash memory;

[0068] Figure 2 is a storage array structure;

[0069] Figure 3 It is a schematic diagram of the processing process of inspection processing and duplicate data deleting processing in traditional technology;

[0070] Figure 4 An application environment diagram of a data processing method in one embodiment;

[0071] Figure 5 is a flow chart of a data processing method in one embodiment;

[0072] Figure 6 is a schematic flow chart of step 503 in another embodiment;

[0073] Figure 7 A schematic diagram of a process for obtaining first inspection data when no data to be processed is read in another embodiment;

[0074] Figure 8 A flowchart diagram of step 503 in another embodiment;

[0075] Fig. 9 A schematic diagram of a process for obtaining identification data in the related art;

[0076] Fig.10 is a schematic flow chart of step 501 in another embodiment;

[0077] Fig.11 is a schematic flow chart of step 1003 in another embodiment;

[0078] Fig.12 is a flowchart of an exemplary data processing method in one embodiment;

[0079] Fig.13 is a structural block diagram of a data processing device in one embodiment;

[0080] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] With the development of storage technology, NAND FLASH is widely used as a storage medium, such as solid-state hard disks. However, NAND FLASH has the characteristic that the data stored in it will be lost over time. Figure 1 , is the relationship between the number of error bits of stored data and the data static time. It can be seen that as the static time increases, the number of error bits also increases. Therefore, some error correction methods are used in flash memory, such as ECC (Error Checking and Correction) technology, to check and repair the data in the flash memory.

[0083] However, ECC error correction technology has a limit on its capabilities, so flash memory also uses consistency check technology to check and repair data.

[0084] Reference Figure 2 , which is a storage array composed of multiple flash memories, wherein each flash memory serves as a storage node. The storage array also includes a control node, which actively reads the storage data in each storage node regularly to perform consistency checks on the storage data in each storage node, and repairs the data if it is wrong.

[0085] In addition, as the amount of stored data increases, duplicate data may be stored in each storage node. In order to increase the capacity of the available space in the flash memory, the control node will also actively read the stored data in each storage node regularly. If the stored data is the same, one copy will be retained and the others will be deleted.

[0086] However, the above data processing methods all require the control node to actively read the storage data in each storage node regularly. Figure 3 ,It can be seen that periodic consistency inspection and ,deletion functions will occupy too much bandwidth resources between the ,control nodes and the storage nodes, resulting in less available bandwidth for the ,host to access the storage nodes through the control nodes, affecting the ,access efficiency of the host to the storage nodes.

[0087] In view of this, the present application provides a data processing method, device, equipment, readable storage medium and program product, by obtaining the opportunity table corresponding to the storage array and performing a read operation on each storage address, a first reading result can be obtained, and according to the first reading result, each data to be processed can be inspected to obtain multiple first inspection data, and duplicate data deletion processing is performed on each storage address according to each identification data and each first inspection data, wherein the opportunity table includes multiple storage addresses and identification data of the data to be processed stored in each storage address, and the multiple storage addresses are storage addresses in at least one storage node in the storage array. In this way, the control node only needs to perform active data reading once according to the storage address stored in the opportunity table, and the inspection processing and duplicate data deletion processing of the data to be processed in each storage address can be completed, avoiding the problem of excessive bandwidth occupation of the control node and each storage node in the traditional technology that the data in the storage node needs to be repeatedly read to complete the inspection and duplicate data deletion processing of the data. The technical solution provided by the present application occupies less bandwidth between the control node and each storage node, provides more bandwidth for the host to access the storage node through the control node, and improves the access efficiency of the host to each storage node.

[0088] The data processing method provided in the embodiment of the present application can be applied to Figure 4 In the application environment shown, a host 401 communicates with a storage array, which includes a control node 402 and multiple storage nodes 403 , wherein the storage node 403 may be a flash memory, and the host 401 may access each storage node 403 through the control node 402 .

[0089] In an exemplary embodiment, Figure 5 As shown, a data processing method is provided, which is applied to Figure 4 The control node in FIG. 1 is taken as an example to illustrate, and the following steps 501 to 503 are included. Among them:

[0090] Step 501: Obtain an opportunity table corresponding to a storage array.

[0091] The opportunity table includes a plurality of storage addresses and identification data of the to-be-processed data stored in each storage address, and the plurality of storage addresses are storage addresses in at least one storage node in the storage array.

[0092] The identification data may be data used to characterize the content of the data to be processed, for example, it may be a hash value of the data to be processed after being processed in a preset hash method, or it may be a code obtained by encoding the data to be processed in other encoding methods.

[0093] Optionally, the control node may obtain the opportunity table from a preset storage space. Optionally, the control node may obtain an externally input opportunity table from other external input devices.

[0094] In the embodiment of the present application, since the control node can periodically process the data in each storage node, the control node can obtain the opportunity table periodically or at a specified time.

[0095] Step 502: Perform a read operation on each storage address to obtain a first read result.

[0096] After acquiring the opportunity table, the control node may perform a read operation according to each storage address in the opportunity table to try to read the data stored in each storage address.

[0097] Regarding the process in which the control node attempts to perform a read operation on each storage address, in an embodiment of the present application, for a storage address, the control node may generate a read instruction based on the storage address, and send the read instruction to the storage node corresponding to the storage address based on the storage address.

[0098] Optionally, if the first read result is a successful read, the control node will obtain the data to be processed returned by the storage node according to the read instruction; optionally, if the first read result is a read failure, the control node cannot obtain the data to be processed returned by the storage node.

[0099] Step 503 , according to the first reading result, perform inspection processing on each data to be processed to obtain a plurality of first inspection data, and perform duplicate data deletion processing on each storage address according to each identification data and each first inspection data.

[0100] In a possible implementation, if the first reading result is a reading failure, the control node cannot obtain the data to be processed, that is, the data to be processed may have an error problem, resulting in the data being unable to be read. Therefore, the control node can perform data repair on the data to be processed during the inspection process and use the repaired data as the first inspection data.

[0101] In another possible implementation, if the first reading result is a successful reading, the control node can obtain the data to be processed, and can perform inspection processing on the data to be processed to obtain the first inspection data. Regarding the process of performing inspection processing on the data to be processed, optionally, if the control node detects that there is erroneous data in the data to be processed according to a preset method, the control node can perform data repair on the data to be processed during the inquiry processing, and use the repaired data as the first inspection data; optionally, if the control node detects that there is no erroneous data in the data to be processed according to a preset method, the control node can directly use the data to be processed as the first inspection data.

[0102] After obtaining the first inspection processing data, the control node can perform duplicate data deduplication processing on each storage address according to the identification data corresponding to each storage address and each first inspection data, thereby deleting duplicate data from each storage address and increasing the available storage space in each storage node.

[0103] In a possible implementation, after the control node completes the inspection of each data to be processed and the deduplication of each storage address, a data processing completion mark can be added to each storage address to avoid repeated data processing of the storage address and reduce unnecessary waste of resources.

[0104] In the above embodiment, the control node only needs to actively read the data once according to the storage address stored in the opportunity table to complete the inspection processing and duplicate data deletion processing of the data to be processed in each storage address, thereby avoiding the problem of excessive bandwidth occupation of the control node and each storage node in the traditional technology. The technical solution provided in the present application occupies less bandwidth between the control node and each storage node, provides more bandwidth for the host to access the storage node through the control node, and improves the access efficiency of the host to each storage node.

[0105] In one embodiment, based on the above Figure 5 The embodiment shown, see Figure 6 This embodiment involves a process of performing inspection processing on each data to be processed according to the first reading result to obtain a plurality of first inspection data. Figure 6 As shown, step 503 may include step 601 and step 602 .

[0106] Step 601: for a storage address, if the first read result includes the to-be-processed data corresponding to the storage address, verify the to-be-processed data according to the first verification code included in the first read result.

[0107] In an embodiment of the present application, for a storage node, if the first read result is a successful read, the control node can obtain the data to be processed corresponding to the storage address included in the first read result. In a possible implementation, the control node can also obtain a first verification code corresponding to the storage address.

[0108] Optionally, the first check code may be a data code used to check the data to be processed. The control node may check the data to be processed according to the first check code, and determine whether there is erroneous data in the data to be processed according to the check result.

[0109] Step 602, obtaining first inspection data according to the verification result.

[0110] In a possible implementation, if the verification result is a verification failure, the control node may obtain metadata corresponding to the storage address, and perform data repair on the data to be processed according to the metadata to obtain the first inspection data.

[0111] In another possible implementation, if the verification result is successful, the control node may determine that the data to be processed is the first inspection data.

[0112] In this way, after successfully reading the data to be processed, the control node can perform inspection processing on the data to be processed by the above method, thereby obtaining correct first inspection data, thereby improving the correctness of the data.

[0113] In one embodiment, based on the above Figure 6 The embodiment shown, see Figure 7 , this embodiment involves a process of obtaining first inspection data without reading the data to be processed. Figure 7 As shown, the process may include step 701 and step 702.

[0114] Step 701: If the first read result does not include the to-be-processed data corresponding to the storage address, then obtain metadata corresponding to the storage address.

[0115] Step 702: Perform data repair on the data to be processed based on the metadata to obtain first inspection data.

[0116] In an embodiment of the present application, for a storage node, if the first read result is a read failure, the first read result will not include the to-be-processed data corresponding to the storage address. The control node can obtain the metadata corresponding to the storage address from the storage node corresponding to the storage address according to the storage address, and perform data repair on the to-be-processed data based on the metadata, and use the repaired data as the first inspection data.

[0117] In the above embodiment, when the data to be processed cannot be read from the storage address, the control node performs inspection processing on the data to be processed, repairs the data that cannot be read due to data errors, and improves the correctness of the data.

[0118] In one embodiment, based on the above Figure 5 The embodiment shown, see Figure 8 This embodiment involves a process of performing data deduplication processing on each storage address according to each identification data and each first inspection data. Figure 8 As shown, step 503 may also include steps 801 to 803.

[0119] Step 801, determining multiple target storage addresses from an opportunity table.

[0120] Among them, multiple target storage addresses correspond to the same identification data.

[0121] In the embodiment of the present application, after the to-be-processed data stored at the storage address in the opportunity table is inspected, the control node may continue to perform a deduplication process on each inspected data.

[0122] In an embodiment of the present application, since the data size of each first inspection data is too large, directly comparing each first inspection data may cause unnecessary waste of resources. Therefore, in a possible implementation manner, since the identification data represents the content of the data stored in the control node, that is, represents the content of the first inspection data, the control node can first compare each identification data. If there are multiple groups of identical identification data, the control node can use the storage addresses corresponding to each group of identical identification data as multiple groups of target storage addresses. For a group of target storage addresses, the control node can perform duplicate data deletion processing on the target first inspection data corresponding to each target storage address.

[0123] Step 802: compare target first inspection data corresponding to multiple target storage addresses.

[0124] For a group of target storage addresses, the control node may obtain the first inspection data corresponding to each target storage address, and compare each first inspection data to determine whether each target first inspection data is the same.

[0125] Regarding the comparison method, optionally, the control node may compare each target first inspection data one by one. Optionally, the control node may input each target first inspection data into a trained comparison model to determine whether the same target first data exists therein.

[0126] Step 803: If the first inspection data of each target is the same, deduplication processing is performed on the target to-be-processed data stored in each target storage address.

[0127] In an embodiment of the present application, if the same target first inspection data exists, the control node can perform duplicate deletion processing on the storage address corresponding to the same target first inspection data. Regarding the processing process, in a possible implementation method, the control node can obtain the static time corresponding to each target data to be processed, and then determine the reserved storage address corresponding to the shortest static time among each static time, and delete the target data to be processed in other target storage addresses except the reserved storage address.

[0128] In this way, the target data to be processed in the reserved storage address corresponding to the shortest static time is retained, and the target data to be processed in other target storage addresses is deleted, so that the target data to be processed finally retained is the latest updated data, ensuring the timeliness of the data.

[0129] In the above embodiment, after each data to be processed is inspected, deduplication processing is continued on the data in each storage address, thereby avoiding the problem of excessive bandwidth occupation caused by multiple active readings of the storage node due to subsequent deduplication processing by the control node, broadening the available bandwidth for the host to access the storage node, and improving the access efficiency of the host to the storage node.

[0130] In one possible implementation, based on Figure 8 In the embodiment, the data processing method may further include: performing deduplication processing on the first inspection data of each target.

[0131] exist Figure 8 In the embodiment shown, the control node deduplicates the data in each storage address. In the embodiment of the present application, the control node also deduplicates the data of each identical target first inspection data, thereby ensuring that the duplicate target first inspection data will not be rewritten to the target storage address where the data has been deleted.

[0132] In a possible implementation, based on the above embodiment, the data processing method may further include: for each first inspection data, if the first inspection data is data repaired, sending the first inspection data to the corresponding storage node along with the first update write instruction.

[0133] The first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first patrol data according to the first patrol data.

[0134] Through the above steps, the control node completes the inspection processing of the data to be processed in each storage address in the opportunity table, as well as the duplicate data deletion processing. However, for the processed first inspection data, the data may be data that has undergone data repair. In an embodiment of the present application, the control node can detect each first inspection data. If there is first inspection data after data repair, the control node can use the first inspection data to update the data to be processed in the corresponding storage address, thereby ensuring the correctness of the data in the storage address.

[0135] In the embodiment of the present application, the opportunity table summarizes the identification data of each storage address and the data to be processed in each storage address. The identification data can represent the content of each data to be processed. When the host actively writes the data to be processed into the corresponding storage node, the control node processes the data to be processed by some preset conversion calculation methods. The process of the control node obtaining the identification data of the data to be processed in the related art can refer to Fig. 9 , including the following steps: the host sends a write instruction, which includes the data to be written to a certain storage address and the storage address. The control node receives the write instruction and puts the data into the cache, and then returns a prompt message of the completion of the write to the host, thereby prompting the host to continue to send write instructions. At the same time, the control node calculates the identification data of each data in the cache, enters the identification data into the opportunity table after the calculation is completed, and then writes the data to the corresponding storage address. In this process, since the prompt message of the completion of the write has been output, the cache may store a large amount of data that requires the calculation of identification data at the same time. When the cache space is insufficient, the identification data may not be calculated in time, resulting in data loss.

[0136] In view of this, based on the above Figure 5 The embodiment shown, see Fig.10 This embodiment involves a process of obtaining an opportunity table corresponding to a storage array, which can greatly alleviate the computing pressure of a control node. Fig.10 As shown, step 501 may also include steps 1001 to 1003.

[0137] Step 1001, obtaining multiple write instructions sent by the host.

[0138] The write instruction includes write data and a write address corresponding to the write data.

[0139] Step 1002: If the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, each write data is written to each write address, and a write success prompt message is returned to the host.

[0140] Step 1003 , when the occupancy rate is less than a preset occupancy rate threshold, obtain identification data corresponding to each write data according to each write address, and obtain an opportunity table according to each identification data.

[0141] In an embodiment of the present application, after receiving multiple write instructions sent by the host, the control node can obtain the write data to be written and the write address corresponding to each write data, and each write address corresponds to a storage space in each storage node.

[0142] After obtaining the write instruction, in order to avoid excessive computing pressure, which results in the write data being put into the memory and then having no time to calculate its identification data, the control node can first detect whether the computing pressure of the cache in the control node exceeds the threshold. The detection method can be to judge by detecting the occupancy rate of the cache. When the occupancy rate is greater than the preset occupancy rate threshold, the control node can determine that the computing pressure of the cache is too high. If the occupancy rate is less than the preset occupancy rate threshold, the control node can determine that the computing pressure of the cache is within the expected range.

[0143] In a possible implementation manner, if the occupancy rate is less than a preset occupancy rate threshold, the control node may Fig. 9 The identification data calculation method provided in the present invention processes the written data to obtain the identification data for entering the opportunity table.

[0144] In another possible implementation, if the occupancy rate is greater than a preset occupancy rate threshold, the control node can directly write each write data into the corresponding write address and return a write success prompt message to the host to prompt the host to continue writing data. For those write data that are directly written to the write address without calculating the identification data, the control node can wait for the appropriate actual calculation of its identification data.

[0145] In an embodiment of the present application, the control node can detect the cache occupancy rate in real time or periodically. When the cache occupancy rate is less than a preset occupancy rate threshold, the control node can calculate the identification data of each write data directly written into the write address based on each write address.

[0146] In the above embodiment, when the cache pressure is too great, the write data is directly written to the address, and a write success prompt message is returned to the host to prompt the host to continue writing data, and then wait for the appropriate actual calculation of the identification data of each write data, which greatly alleviates the calculation pressure of the cache and improves the calculation efficiency of the identification data.

[0147] Based on the above Fig.10 The embodiment shown, see Fig.11 This embodiment relates to a process of obtaining identification data corresponding to each written data according to each written address. Fig.11 As shown, step 1003 may also include steps 1101 to 1103.

[0148] Step 1101, performing a read operation on each write address to obtain a second read result.

[0149] When the control node detects that the occupancy rate of the cache is less than a preset occupancy rate threshold, it may start processing the write data corresponding to each write address.

[0150] In an embodiment of the present application, the control node first attempts to read according to each write address. If the read is successful, the second read result includes the write data. If the read fails, the second read result does not include the write data.

[0151] Step 1102: Perform inspection processing on each write data according to the second reading result to obtain a plurality of second inspection data.

[0152] It is understandable that the control node detects that the actual possible time for the cache occupancy rate to be less than the preset occupancy rate threshold is the time process for writing the write data directly to the write address, and erroneous data may have appeared in the write data. Therefore, before calculating the identification data corresponding to each write data, the control node needs to first inspect each write data to ensure the correctness of the identification data.

[0153] Regarding the process in which the control node performs patrol processing on each write data according to the second read result, in a possible implementation, for a write address, optionally, if the second read result does not include the write data corresponding to the write address (that is, the control node cannot successfully read the write data), the control node may perform data repair on the write data according to the metadata corresponding to the write address to obtain second patrol data; optionally, if the second read result includes the write data corresponding to the write address, the control node verifies the data to be processed according to the second verification code included in the second read result, and if the verification result is a verification failure (indicating that the write data read from the write address is incorrect), the control node performs data repair on the write data according to the metadata corresponding to the write address to obtain second patrol data, and if the verification result is a verification success (indicating that the write data read from the write address is correct), the control node determines that the write data is the second patrol data, wherein the second patrol data is correct data.

[0154] By performing inspection processing on each written data in the above manner, the correctness of the identification data and the correctness of the data in each written address can be ensured.

[0155] Step 1103: Process each second inspection data according to a preset condition to obtain identification data corresponding to each write data.

[0156] After obtaining multiple correct second inspection data, the control node can process each second inspection data according to preset conditions to obtain identification data corresponding to each write data. Regarding the processing process, optionally, the control node can hash each second inspection data according to a preset hash method, and use the obtained hash value as the identification data of each write data corresponding to each second inspection data; optionally, the control node can input each second inspection data into a preset encoding conversion model to obtain identification data corresponding to each write data output by the encoding conversion model.

[0157] In the above embodiment, each written data is firstly inspected, and then the identification data corresponding to each written data is calculated using the second inspection data obtained by the inspection, thereby ensuring the correctness of the identification data.

[0158] Based on the above embodiments, the embodiments of the present application also involve a process of updating data at each write address using each second patrol data after obtaining the identification data, specifically including: for each second patrol data, if the second patrol data is data repaired, the control node sends the second patrol data carrying a second update write instruction to the corresponding storage node.

[0159] The second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0160] In this way, the write data in the write address is updated by using the second inspection data that has been inspected, thereby improving the correctness of the data in the write address.

[0161] In one embodiment, referring to Fig.12 , provides an exemplary data processing method, which can be applied to Figure 1 The control node in .

[0162] Step 1201, obtaining multiple write instructions sent by the host.

[0163] The write instruction includes write data and a write address corresponding to the write data.

[0164] Step 1202: If the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, each write data is written to each write address, and a write success prompt message is returned to the host.

[0165] Step 1203: When the occupancy rate is less than a preset occupancy rate threshold, a read operation is performed on each write address to obtain a second read result.

[0166] Step 1204 , for a write address, if the second read result does not include the write data corresponding to the write address, then data repair is performed on the write data according to the metadata corresponding to the write address to obtain second inspection data.

[0167] Step 1205: If the second read result includes the write data corresponding to the write address, verify the data to be processed according to the second verification code included in the second read result.

[0168] Step 1206: If the verification result is a verification failure, data repair is performed on the written data according to the metadata corresponding to the written address to obtain second inspection data.

[0169] Step 1207: If the verification result is successful, determine that the written data is the second inspection data.

[0170] Step 1208: Process each second inspection data according to preset conditions to obtain identification data corresponding to each write data.

[0171] Step 1209: Obtain an opportunity table according to each identification data.

[0172] Step 1210 : for each second inspection data, if the second inspection data is data that has been repaired, the second inspection data is carried in a second update write instruction and sent to a corresponding storage node.

[0173] The second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0174] The opportunity table includes a plurality of storage addresses and identification data of the to-be-processed data stored in each storage address, and the plurality of storage addresses are storage addresses in at least one storage node in the storage array.

[0175] Step 1211, performing a read operation on each storage address to obtain a first read result.

[0176] Step 1212: for a storage address, if the first read result includes the to-be-processed data corresponding to the storage address, verify the to-be-processed data according to the first verification code included in the first read result.

[0177] Step 1213: If the verification result is a verification failure, metadata corresponding to the storage address is obtained, and data repair is performed on the data to be processed according to the metadata to obtain first inspection data.

[0178] Step 1214: If the verification result is successful, determine that the data to be processed is the first inspection data.

[0179] Step 1215: If the first read result does not include the to-be-processed data corresponding to the storage address, then obtain metadata corresponding to the storage address.

[0180] Step 1216: Perform data repair on the data to be processed based on the metadata to obtain first inspection data.

[0181] Step 1217, determining multiple target storage addresses from the opportunity table, the multiple target storage addresses corresponding to the same identification data.

[0182] Step 1218: compare the target first inspection data corresponding to the multiple target storage addresses.

[0183] Step 1219: If the first inspection data of each target is the same, the static time corresponding to the to-be-processed data of each target is obtained.

[0184] Step 1220: Determine the reserved storage address corresponding to the shortest static time among the static times.

[0185] Step 1221, deleting the target data to be processed in the target storage addresses other than the reserved storage addresses in each target storage address.

[0186] Step 1222, deduplication processing is performed on the first inspection data of each target.

[0187] Step 1223: for each first inspection data, if the first inspection data is data that has been repaired, the first inspection data is carried in the first update write instruction and sent to the corresponding storage node.

[0188] The first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first patrol data according to the first patrol data.

[0189] Step 1224, adding a data processing completion mark to each storage address.

[0190] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0191] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0192] In an exemplary embodiment, Fig.13 As shown, a data processing device is provided, which is used for a control node in a storage array, including: an acquisition module 1301, a reading module 1302 and a processing module 1303, wherein:

[0193] An acquisition module 1301 is used to acquire an opportunity table corresponding to a storage array, wherein the opportunity table includes multiple storage addresses and identification data of to-be-processed data stored in each storage address, and the multiple storage addresses are storage addresses in at least one storage node in the storage array;

[0194] The reading module 1302 is used to perform a reading operation on each storage address to obtain a first reading result;

[0195] The processing module 1303 is used to perform inspection processing on each to-be-processed data according to the first reading result to obtain a plurality of first inspection data, and perform duplicate data deletion processing on each storage address according to each identification data and each first inspection data.

[0196] In one embodiment, the processing module 1303 includes:

[0197] a verification unit, configured to verify, for a storage address, the data to be processed according to a first verification code included in the first reading result, if the first reading result includes the data to be processed corresponding to the storage address;

[0198] The inspection unit is used to obtain first inspection data according to the verification result.

[0199] In one embodiment, the inspection unit is specifically used to perform:

[0200] If the verification result is a verification failure, metadata corresponding to the storage address is obtained, and data repair is performed on the data to be processed according to the metadata to obtain first inspection data;

[0201] If the verification result is successful, it is determined that the data to be processed is the first inspection data.

[0202] In one embodiment, the processing module 1303 further includes:

[0203] a metadata acquisition unit, configured to acquire metadata corresponding to the storage address if the first reading result does not include the to-be-processed data corresponding to the storage address;

[0204] The repair unit is used to perform data repair on the data to be processed based on the metadata to obtain first inspection data.

[0205] In one embodiment, the processing module 1303 further includes:

[0206] A target determination unit, used to determine a plurality of target storage addresses from the opportunity table, wherein the plurality of target storage addresses correspond to the same identification data;

[0207] A comparison unit, used for comparing target first inspection data corresponding to a plurality of target storage addresses;

[0208] The address deleting unit is used for deleting duplicate data of the target to-be-processed data stored in each target storage address if the first inspection data of each target are the same.

[0209] In one embodiment, the address deletion unit is specifically configured to perform:

[0210] Obtain the static time corresponding to each target data to be processed;

[0211] Determine a reserved storage address corresponding to the shortest static time among the static times;

[0212] The target data to be processed in the target storage addresses other than the reserved storage addresses in each target storage address are deleted.

[0213] In one embodiment, the processing module 1303 further includes:

[0214] The inspection data deleting unit is used to perform a deduplication process on each target first inspection data.

[0215] In one embodiment, the inspection data deletion unit is specifically used to execute:

[0216] For each first inspection data, if the first inspection data is data-repaired data, the first inspection data is sent to the corresponding storage node with a first update write instruction, and the first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first inspection data according to the first inspection data.

[0217] In one embodiment, the data processing device further comprises:

[0218] The marking module is used to add a data processing completion mark to each storage address.

[0219] In one embodiment, the acquisition module 1301 includes:

[0220] A write instruction acquisition unit, used for acquiring a plurality of write instructions sent by a host, wherein the write instructions include write data and a write address corresponding to the write data;

[0221] The proportion greater than unit is used to write each write data to each write address if the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, and return a write success prompt message to the host;

[0222] The less than proportion unit is used to obtain identification data corresponding to each write data according to each write address when the occupancy rate is less than a preset occupancy rate threshold, and obtain an opportunity table according to each identification data.

[0223] In one embodiment, the proportion less than unit is specifically used to perform:

[0224] Performing a read operation on each write address to obtain a second read result;

[0225] According to the second reading result, each written data is inspected to obtain a plurality of second inspection data;

[0226] Each second inspection data is processed according to a preset condition to obtain identification data corresponding to each write data.

[0227] In one embodiment, the less than unit is further configured to perform:

[0228] For a write address, if the second read result does not include the write data corresponding to the write address, then data repair is performed on the write data according to the metadata corresponding to the write address to obtain second inspection data;

[0229] If the second read result includes write data corresponding to the write address, the data to be processed is verified according to the second verification code included in the second read result. If the verification result is a verification failure, the write data is repaired according to the metadata corresponding to the write address to obtain second inspection data. If the verification result is a verification success, it is determined that the write data is the second inspection data.

[0230] In one embodiment, the less than unit is further configured to perform:

[0231] For each second inspection data, if the second inspection data is data repaired, the second inspection data is carried in a second update write instruction and sent to the corresponding storage node, and the second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0232] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0233] In an exemplary embodiment, a computer device is provided. The computer device may be a control node in a storage array. The internal structure diagram of the computer device may be as shown in FIG. Fig.14As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0234] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0235] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0236] Acquire an opportunity table corresponding to the storage array, wherein the opportunity table includes a plurality of storage addresses and identification data of to-be-processed data stored in each of the storage addresses, wherein the plurality of storage addresses are storage addresses in at least one storage node in the storage array;

[0237] Performing a read operation on each of the storage addresses to obtain a first read result;

[0238] According to the first reading result, each of the to-be-processed data is subjected to inspection processing to obtain a plurality of first inspection data, and according to each of the identification data and each of the first inspection data, duplicate data deletion processing is performed on each of the storage addresses.

[0239] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0240] For one of the storage addresses, if the first reading result includes the data to be processed corresponding to the storage address, verifying the data to be processed according to a first verification code included in the first reading result;

[0241] The first inspection data is obtained according to the verification result.

[0242] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0243] If the verification result is a verification failure, obtaining metadata corresponding to the storage address, and performing data repair on the data to be processed according to the metadata to obtain the first inspection data;

[0244] If the verification result is successful, it is determined that the data to be processed is the first inspection data.

[0245] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0246] If the first reading result does not include the to-be-processed data corresponding to the storage address, obtaining metadata corresponding to the storage address;

[0247] The data to be processed is repaired based on the metadata to obtain the first inspection data.

[0248] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0249] Determine a plurality of target storage addresses from the opportunity table, the plurality of target storage addresses corresponding to the same identification data;

[0250] comparing the target first inspection data corresponding to the plurality of target storage addresses;

[0251] If the first inspection data of each target are the same, deduplication processing is performed on the target to-be-processed data stored in each target storage address.

[0252] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0253] Obtaining the static time corresponding to each of the target data to be processed;

[0254] Determine a reserved storage address corresponding to the shortest static time among the static times;

[0255] The target data to be processed in the target storage addresses other than the reserved storage address in the target storage addresses are deleted.

[0256] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0257] De-duplication processing is performed on each of the target first inspection data.

[0258] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0259] For each of the first inspection data, if the first inspection data is data-repaired data, the first inspection data is carried in a first update write instruction and sent to the corresponding storage node, and the first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first inspection data according to the first inspection data.

[0260] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0261] A data processing completion mark is added to each of the storage addresses.

[0262] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0263] Acquire multiple write instructions sent by the host, wherein the write instructions include write data and write addresses corresponding to the write data;

[0264] If the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, each of the write data is written to each of the write addresses, and a write success prompt message is returned to the host;

[0265] In a case where the occupancy rate is less than the preset occupancy rate threshold, identification data corresponding to each of the write data is acquired according to each of the write addresses, and the opportunity table is acquired according to each of the identification data.

[0266] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0267] Performing a read operation on each of the write addresses to obtain a second read result;

[0268] According to the second reading result, performing inspection processing on each of the written data to obtain a plurality of second inspection data;

[0269] Each of the second inspection data is processed according to a preset condition to obtain identification data corresponding to each of the write data.

[0270] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0271] For one of the write addresses, if the second reading result does not include the write data corresponding to the write address, then performing data repair on the write data according to the metadata corresponding to the write address to obtain the second inspection data;

[0272] If the second read result includes the write data corresponding to the write address, the data to be processed is verified according to the second verification code included in the second read result. If the verification result is a verification failure, the write data is repaired according to the metadata corresponding to the write address to obtain the second inspection data. If the verification result is a verification success, it is determined that the write data is the second inspection data.

[0273] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0274] For each second inspection data, if the second inspection data is data-repaired data, the second inspection data is carried in a second update write instruction and sent to the corresponding storage node, and the second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0275] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0276] Acquire an opportunity table corresponding to the storage array, wherein the opportunity table includes a plurality of storage addresses and identification data of to-be-processed data stored in each of the storage addresses, wherein the plurality of storage addresses are storage addresses in at least one storage node in the storage array;

[0277] Performing a read operation on each of the storage addresses to obtain a first read result;

[0278] According to the first reading result, each of the to-be-processed data is subjected to inspection processing to obtain a plurality of first inspection data, and according to each of the identification data and each of the first inspection data, duplicate data deletion processing is performed on each of the storage addresses.

[0279] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0280] For one of the storage addresses, if the first reading result includes the data to be processed corresponding to the storage address, verifying the data to be processed according to a first verification code included in the first reading result;

[0281] The first inspection data is obtained according to the verification result.

[0282] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0283] If the verification result is a verification failure, obtaining metadata corresponding to the storage address, and performing data repair on the data to be processed according to the metadata to obtain the first inspection data;

[0284] If the verification result is successful, it is determined that the data to be processed is the first inspection data.

[0285] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0286] If the first reading result does not include the to-be-processed data corresponding to the storage address, obtaining metadata corresponding to the storage address;

[0287] The data to be processed is repaired based on the metadata to obtain the first inspection data.

[0288] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0289] Determine a plurality of target storage addresses from the opportunity table, the plurality of target storage addresses corresponding to the same identification data;

[0290] comparing the target first inspection data corresponding to the plurality of target storage addresses;

[0291] If the first inspection data of each target are the same, deduplication processing is performed on the target to-be-processed data stored in each target storage address.

[0292] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0293] Obtaining the static time corresponding to each of the target data to be processed;

[0294] Determine a reserved storage address corresponding to the shortest static time among the static times;

[0295] The target data to be processed in the target storage addresses other than the reserved storage address in the target storage addresses are deleted.

[0296] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0297] De-duplication processing is performed on each of the target first inspection data.

[0298] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0299] For each of the first inspection data, if the first inspection data is data-repaired data, the first inspection data is carried in a first update write instruction and sent to the corresponding storage node, and the first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first inspection data according to the first inspection data.

[0300] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0301] A data processing completion mark is added to each of the storage addresses.

[0302] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0303] Acquire multiple write instructions sent by the host, wherein the write instructions include write data and write addresses corresponding to the write data;

[0304] If the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, each of the write data is written to each of the write addresses, and a write success prompt message is returned to the host;

[0305] In a case where the occupancy rate is less than the preset occupancy rate threshold, identification data corresponding to each of the write data is acquired according to each of the write addresses, and the opportunity table is acquired according to each of the identification data.

[0306] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0307] Performing a read operation on each of the write addresses to obtain a second read result;

[0308] According to the second reading result, performing inspection processing on each of the written data to obtain a plurality of second inspection data;

[0309] Each of the second inspection data is processed according to a preset condition to obtain identification data corresponding to each of the write data.

[0310] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0311] For one of the write addresses, if the second reading result does not include the write data corresponding to the write address, then performing data repair on the write data according to the metadata corresponding to the write address to obtain the second inspection data;

[0312] If the second read result includes the write data corresponding to the write address, the data to be processed is verified according to the second verification code included in the second read result. If the verification result is a verification failure, the write data is repaired according to the metadata corresponding to the write address to obtain the second inspection data. If the verification result is a verification success, it is determined that the write data is the second inspection data.

[0313] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0314] For each second inspection data, if the second inspection data is data-repaired data, the second inspection data is carried in a second update write instruction and sent to the corresponding storage node, and the second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0315] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0316] Acquire an opportunity table corresponding to the storage array, wherein the opportunity table includes a plurality of storage addresses and identification data of to-be-processed data stored in each of the storage addresses, wherein the plurality of storage addresses are storage addresses in at least one storage node in the storage array;

[0317] Performing a read operation on each of the storage addresses to obtain a first read result;

[0318] According to the first reading result, each of the to-be-processed data is subjected to inspection processing to obtain a plurality of first inspection data, and according to each of the identification data and each of the first inspection data, duplicate data deletion processing is performed on each of the storage addresses.

[0319] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0320] For one of the storage addresses, if the first reading result includes the data to be processed corresponding to the storage address, verifying the data to be processed according to a first verification code included in the first reading result;

[0321] The first inspection data is obtained according to the verification result.

[0322] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0323] If the verification result is a verification failure, obtaining metadata corresponding to the storage address, and performing data repair on the data to be processed according to the metadata to obtain the first inspection data;

[0324] If the verification result is successful, it is determined that the data to be processed is the first inspection data.

[0325] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0326] If the first reading result does not include the to-be-processed data corresponding to the storage address, obtaining metadata corresponding to the storage address;

[0327] The data to be processed is repaired based on the metadata to obtain the first inspection data.

[0328] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0329] Determine a plurality of target storage addresses from the opportunity table, the plurality of target storage addresses corresponding to the same identification data;

[0330] comparing the target first inspection data corresponding to the plurality of target storage addresses;

[0331] If the first inspection data of each target are the same, deduplication processing is performed on the target to-be-processed data stored in each target storage address.

[0332] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0333] Obtaining the static time corresponding to each of the target data to be processed;

[0334] Determine a reserved storage address corresponding to the shortest static time among the static times;

[0335] The target data to be processed in the target storage addresses other than the reserved storage address in the target storage addresses are deleted.

[0336] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0337] De-duplication processing is performed on each of the target first inspection data.

[0338] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0339] For each of the first inspection data, if the first inspection data is data-repaired data, the first inspection data is carried in a first update write instruction and sent to the corresponding storage node, and the first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first inspection data according to the first inspection data.

[0340] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0341] A data processing completion mark is added to each of the storage addresses.

[0342] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0343] Acquire multiple write instructions sent by the host, wherein the write instructions include write data and write addresses corresponding to the write data;

[0344] If the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, each of the write data is written to each of the write addresses, and a write success prompt message is returned to the host;

[0345] In a case where the occupancy rate is less than the preset occupancy rate threshold, identification data corresponding to each of the write data is acquired according to each of the write addresses, and the opportunity table is acquired according to each of the identification data.

[0346] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0347] Performing a read operation on each of the write addresses to obtain a second read result;

[0348] According to the second reading result, performing inspection processing on each of the written data to obtain a plurality of second inspection data;

[0349] Each of the second inspection data is processed according to a preset condition to obtain identification data corresponding to each of the write data.

[0350] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0351] For one of the write addresses, if the second reading result does not include the write data corresponding to the write address, then performing data repair on the write data according to the metadata corresponding to the write address to obtain the second inspection data;

[0352] If the second read result includes the write data corresponding to the write address, the data to be processed is verified according to the second verification code included in the second read result. If the verification result is a verification failure, the write data is repaired according to the metadata corresponding to the write address to obtain the second inspection data. If the verification result is a verification success, it is determined that the write data is the second inspection data.

[0353] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0354] For each second inspection data, if the second inspection data is data-repaired data, the second inspection data is carried in a second update write instruction and sent to the corresponding storage node, and the second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

[0355] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0356] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0357] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0358] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: For use with a control node in a storage array, the method comprises: Acquire an opportunity table corresponding to the storage array, wherein the opportunity table includes a plurality of storage addresses and identification data of to-be-processed data stored in each of the storage addresses, wherein the plurality of storage addresses are storage addresses in at least one storage node in the storage array; Performing a read operation on each of the storage addresses to obtain a first read result; According to the first reading result, each of the to-be-processed data is subjected to inspection processing to obtain a plurality of first inspection data, and according to each of the identification data and each of the first inspection data, duplicate data deletion processing is performed on each of the storage addresses.

2. The method according to claim 1, characterized in that The method of performing inspection processing on each of the to-be-processed data according to the first reading result to obtain a plurality of first inspection data includes: For one of the storage addresses, if the first reading result includes the data to be processed corresponding to the storage address, verifying the data to be processed according to a first verification code included in the first reading result; The first inspection data is obtained according to the verification result.

3. The method according to claim 2, characterized in that The obtaining the first inspection data according to the verification result includes: If the verification result is a verification failure, obtaining metadata corresponding to the storage address, and performing data repair on the data to be processed according to the metadata to obtain the first inspection data; If the verification result is successful, it is determined that the data to be processed is the first inspection data.

4. The method according to claim 2, characterized in that: The method further comprises: If the first reading result does not include the to-be-processed data corresponding to the storage address, obtaining metadata corresponding to the storage address; The data to be processed is repaired based on the metadata to obtain the first inspection data.

5. The method according to claim 1, characterized in that: The performing deduplication processing on each storage address according to each identification data and each first inspection data includes: Determine a plurality of target storage addresses from the opportunity table, the plurality of target storage addresses corresponding to the same identification data; comparing the target first inspection data corresponding to the plurality of target storage addresses; If the first inspection data of each target are the same, deduplication processing is performed on the target to-be-processed data stored in each target storage address.

6. The method according to claim 5, characterized in that The deduplication processing of the target to-be-processed data stored in each of the target storage addresses includes: Obtaining the static time corresponding to each of the target data to be processed; Determine a reserved storage address corresponding to the shortest static time among the static times; The target data to be processed in the target storage addresses other than the reserved storage address in the target storage addresses are deleted.

7. The method according to claim 5, characterized in that The method further comprises: De-duplication processing is performed on each of the target first inspection data.

8. The method according to claim 7, characterized in that The method further comprises: For each of the first inspection data, if the first inspection data is data-repaired data, the first inspection data is carried in a first update write instruction and sent to the corresponding storage node, and the first update write instruction is used to instruct to update the data to be processed in the storage address corresponding to the first inspection data according to the first inspection data.

9. The method according to claim 1, characterized in that: The method further comprises: A data processing completion mark is added to each of the storage addresses.

10. The method according to claim 1, characterized in that The obtaining the opportunity table corresponding to the storage array includes: Acquire multiple write instructions sent by the host, wherein the write instructions include write data and write addresses corresponding to the write data; If the occupancy rate of the cache in the control node is greater than a preset occupancy rate threshold, each of the write data is written to each of the write addresses, and a write success prompt message is returned to the host; In a case where the occupancy rate is less than the preset occupancy rate threshold, identification data corresponding to each of the write data is acquired according to each of the write addresses, and the opportunity table is acquired according to each of the identification data.

11. The method according to claim 10, characterized in that The acquiring identification data corresponding to each of the written data according to each of the written addresses comprises: Performing a read operation on each of the write addresses to obtain a second read result; According to the second reading result, performing inspection processing on each of the written data to obtain a plurality of second inspection data; Each of the second inspection data is processed according to a preset condition to obtain identification data corresponding to each of the write data.

12. The method according to claim 11, characterized in that The method of performing inspection processing on each of the written data according to the second reading result to obtain a plurality of second inspection data includes: For one of the write addresses, if the second reading result does not include the write data corresponding to the write address, then performing data repair on the write data according to the metadata corresponding to the write address to obtain the second inspection data; If the second read result includes the write data corresponding to the write address, the data to be processed is verified according to the second verification code included in the second read result. If the verification result is a verification failure, the write data is repaired according to the metadata corresponding to the write address to obtain the second inspection data. If the verification result is a verification success, it is determined that the write data is the second inspection data.

13. The method according to claim 11, characterized in that The method further comprises: For each second inspection data, if the second inspection data is data-repaired data, the second inspection data is carried in a second update write instruction and sent to the corresponding storage node, and the second update write instruction is used to instruct to update the write data in the write address according to the second inspection data.

14. A data processing device, characterized in that: For use in a control node in a storage array, the device comprises: an acquisition module, configured to acquire an opportunity table corresponding to the storage array, wherein the opportunity table includes a plurality of storage addresses and identification data of the to-be-processed data stored in each of the storage addresses, wherein the plurality of storage addresses are storage addresses in at least one storage node in the storage array; A reading module, used for performing a reading operation on each of the storage addresses to obtain a first reading result; The processing module is used to perform inspection processing on each of the to-be-processed data according to the first reading result to obtain a plurality of first inspection data, and perform duplicate data deletion processing on each of the storage addresses according to each of the identification data and each of the first inspection data.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images