Data processing method and device, equipment, storage medium and computer program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a computer program product, and is applied to a distributed storage system.The method comprises the steps that when it is detected that a first abnormal sector appears in a first hard disk in the distributed storage system, the first abnormal sector is generated in the first hard disk; obtaining capacity information of the first abnormal sector and / or quantity information of discontinuous sectors in the first abnormal sector; under the condition that the capacity information is smaller than or equal to a first value and / or the quantity information is smaller than or equal to a second value, shielding the first abnormal sector, and searching a copy sector of the first abnormal sector in the distributed storage system; replica data in the replica sector is copied to other sectors except the first abnormal sector in the first hard disk, so that replica data of the first abnormal sector is accessed by accessing other sectors. And resource waste can be reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium, and computer program product. Background Art

[0002] In addition to hard drives reaching the end of their lifespan or becoming damaged and unusable, partial failures can also occur due to hardware quality, environmental factors, and upper-layer software usage. Distributed storage systems typically handle hard drive failures by simply replacing them with new ones, but this approach wastes resources. Summary of the Invention

[0003] Embodiments of the present application provide a data processing method, apparatus, device, storage medium, and computer program product that can reduce resource waste.

[0004] To achieve the above objectives, the technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, the present application provides a data processing method, applied to a distributed storage system, the method comprising:

[0006] When a first abnormal sector is detected in a first hard disk in the distributed storage system, obtaining capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector;

[0007] When the capacity information is less than or equal to a first value, and / or the quantity information is less than or equal to a second value, shielding the first abnormal sector and searching for a replica sector of the first abnormal sector in the distributed storage system;

[0008] The copy data in the copy sector is copied to other sectors of the first hard disk except the first abnormal sector, so that the copy data of the first abnormal sector can be accessed by accessing other sectors.

[0009] In a second aspect, the present application proposes a data processing device, applied to a distributed storage system, comprising:

[0010] an acquiring unit, configured to, upon detecting that a first abnormal sector appears on a first hard disk in the distributed storage system, acquire capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector;

[0011] a shielding unit, configured to shield the first abnormal sector and search for a replica sector of the first abnormal sector in the distributed storage system when the capacity information is less than or equal to a first value and / or the quantity information is less than or equal to a second value;

[0012] The copy unit is configured to copy the copy data in the copy sector to other sectors of the first hard disk except the first abnormal sector, so as to access the copy data of the first abnormal sector by accessing other sectors.

[0013] In a third aspect, the present application proposes a data processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0014] In a fourth aspect, the present application proposes a storage medium on which a computer program is stored, which implements the steps of any of the above methods when executed by a processor.

[0015] In a fifth aspect, the present application proposes a computer program product, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0016] The present application proposes a data processing method, apparatus, device, storage medium and computer program product, which are applied to a distributed storage system. The method includes: when a first abnormal sector is detected in a first hard disk in the distributed storage system, obtaining capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector; when the capacity information is less than or equal to a first value, and / or the number information is less than or equal to a second value, shielding the first abnormal sector and searching for a copy sector of the first abnormal sector in the distributed storage system; copying the copy data in the copy sector to other sectors of the first hard disk except the first abnormal sector, so as to access the copy data of the first abnormal sector by accessing other sectors. By adopting the above-mentioned implementation method, when a first abnormal sector appears on the first hard disk in the distributed storage system, the capacity information of the first abnormal sector and / or the number information of the discontinuous sectors in the first abnormal sector are obtained. When the capacity information is less than or equal to the first value and / or the number information is less than or equal to the second value, the first abnormal sector is shielded, and the data of the copy sector of the first abnormal sector is copied to other sectors in the first hard disk except the first abnormal sector. In this way, when a sector in the first hard disk is partially damaged (i.e., the capacity information is less than or equal to the first value and / or the number information is less than or equal to the second value), the data of the corresponding copy sector is directly used to repair the first abnormal sector, so as to access the first hard disk normally and avoid directly replacing a new hard disk, thereby reducing resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic structural diagram of an exemplary general method for handling hard disk sector damage provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a general method for handling hard disk sector damage provided in an exemplary embodiment of the present application;

[0019] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0020] Figure 4 A schematic structural diagram of an exemplary hard disk sector damage optimization solution provided in an embodiment of the present application;

[0021] Figure 5 A flowchart of an exemplary hard disk sector damage optimization solution identification process provided in an embodiment of the present application;

[0022] Figure 6 A flowchart of an exemplary hard disk sector damage alarm and event recording process provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of an exemplary process of hard disk sector damage provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0025] Figure 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0028] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first," "second," and the like in the embodiments of the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first," "second," and the like may interchange specific orders or precedences where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0029] In addition to the hard drive reaching the end of its service life or being damaged and unusable, the hard drive may also experience partial failure due to hardware quality, environmental factors, and upper-layer software usage. For example:

[0030] (1) Bad tracks and sectors caused by local differences between the hard disk platter and flash memory particles;

[0031] (2) Unstable system power supply leads to power outages and voltage fluctuations that damage the hard disk;

[0032] (3) Hardware and environmental influences such as sector damage or data corruption caused by dust and static electricity;

[0033] (4) The upper-level software uses the hard disk in a way that causes sector damage, such as frequent reading and writing of local locations.

[0034] Hardware, environmental, and software factors can increase the probability of hard drive sector failure by up to 100 times. Current enterprise-level distributed storage clusters optimize hard drive load through software monitoring, load balancing, and pre-flush caching. Environmental and deployment considerations include preventing the effects of power, static electricity, and dust, and using high-quality enterprise-level storage disks. Furthermore, distributed storage software systems generally support data redundancy and data protection strategies to ensure data security and reliability after hard drive sector failure.

[0035] Even after implementing the aforementioned measures in a normal enterprise-level distributed storage cluster environment, with high-quality enterprise-level hard drives, adapted operating environments, and reasonable loads, the probability of drive sector failure is approximately 0.5%-2% per year. However, in harsh operating environments, such as those with high temperature, high humidity, high dust levels, and extremely high loads, the probability of sector failure can rise dramatically, reaching 10% or even higher per year. Under normal conditions and reasonable loads, a small, three-node cluster will have over 100 hard drives, making drive sector failure an inevitable annual occurrence in distributed storage clusters.

[0036] In data centers, high-capacity hard disk drives (HDDs) with a capacity of 20 terabytes (TB) and solid-state drives (SSDs) with a capacity of 7.86 TB are widely used. The capacity of hard disk sectors damaged is generally below the megabyte (MB) level. The current distributed storage cluster processing method is to identify the hard disk with damaged sectors and replace it with a new one. Then, the data is restored on the new disk through the copy data of the data redundancy strategy. This processing method has the problems of long recovery time and waste of resources of the hard disk with damaged sectors. The following takes the redundancy strategy of three copies as an example. Figure 1 A schematic structural diagram of a common method for processing hard disk sector damage provided in an embodiment of the present application; Figure 1 As shown, there are storage server nodes 1, 2, and 3 on the distributed storage system. File A data is stored in three copies on the distributed storage system (i.e., file A data is stored on storage server nodes 1, 2, and 3). On storage server node 1, data A falls on the damaged sector of the hard disk. A bad sector is detected in the hard disk area where data A is located, and a disk failure alarm is generated. Waiting to use a new hard disk to replace the faulty hard disk, the new hard disk is replaced and the hard disk management service is started. The data of the new hard disk is fully reconstructed and restored from other copies. For ease of understanding, Figure 2 A flowchart of a general method for processing hard disk sector damage provided by an embodiment of the present application is provided; Figure 2 The specific steps are as follows:

[0037] 1. Data consistency scan client input and output (IO) access.

[0038] 2. Hard disk access IO.

[0039] 3. Whether to return input and output error (Error Input Output, EIO).

[0040] It should be noted that, if EIO is returned, step 4 is executed.

[0041] 4. Determine whether it is a bad sector.

[0042] It should be noted that, if a bad sector is detected, step 6 is executed.

[0043] 5. Check the hard disk sectors regularly.

[0044] It should be noted that step 4 is performed after step 5.

[0045] 6. A disk failure alarm occurs.

[0046] 7. Wait for the disk swap operation.

[0047] 8. The faulty hard drive enters operation and maintenance mode.

[0048] 9. Remove the faulty disk.

[0049] 10. Wait for manual replacement with a new disk.

[0050] 11. Exit the hard disk operation and maintenance mode.

[0051] 12. Format the hard disk, add it to the cluster, and start the service.

[0052] 13. The cluster rebuilds and restores the new disk data through redundant copies.

[0053] The above solution has the disadvantages of long recovery time and waste of hard disk resources due to damaged sectors.

[0054] Based on this, the embodiment of the present application provides a data processing method, which is applied to a distributed storage system. Figure 3 A flow chart of a data processing method provided in an embodiment of the present application; Figure 3 As shown, the method is as follows:

[0055] S301. When a first abnormal sector is detected in a first hard disk in a distributed storage system, obtain capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector.

[0056] It should be noted that, in practical applications, a distributed storage system can also be understood as a distributed storage cluster.

[0057] It should be noted that the number of hard disks in the distributed storage system can be determined based on actual conditions and is not limited here. The first hard disk can be understood as any hard disk in the distributed storage system. The specific selection can be determined based on actual conditions and is not limited here. The first abnormal sector can be understood as a damaged sector in the first hard disk. In actual applications, the first abnormal sector can be specifically understood as a sector in the first hard disk with hardware damage. The capacity information can be understood as the total capacity information of the sector in the first hard disk with hardware damage (i.e., the first abnormal sector). The quantity information can be understood as the quantity information of the discontinuous damaged sectors in the sector in the first hard disk with hardware damage (i.e., the first abnormal sector).

[0058] In an embodiment of the present application, the process of obtaining the first abnormal sector specifically includes: obtaining access data corresponding to the sector of the first hard disk within a first time; searching for first data representing the access abnormality from the access data; determining the sector corresponding to the first data; and determining the first abnormal sector based on the sector corresponding to the first data.

[0059] It should be noted that the first time can be understood as a certain period of time. In actual applications, the first time can be understood as periodic. Access data can be understood as hard disk access sector IO data. The number of access data can be one or more, and the specific number can be determined according to actual conditions, which is not limited here. The first data representing the access abnormality can be understood as the hard disk accessing sector IO to obtain EIO. The number of first data can be one or more, and the specific number can be determined according to actual conditions, which is not limited here. The sector corresponding to the first data can be understood as a damaged sector in the first hard disk. In actual applications, the sector corresponding to the first data can also be specifically understood as a sector in the first hard disk where hardware damage occurs. The number of sectors corresponding to the first data can be one or more, and the specific number can be determined according to actual conditions, which is not limited here.

[0060] For ease of understanding, the above solution is described here by way of example. Sector corresponding to first data returned by EIO in hard disk access IO among multiple sectors of the first hard disk is regularly obtained, and the sector corresponding to the first data is taken as the first abnormal sector.

[0061] In an embodiment of the present application, the process of determining the first abnormal sector based on the sector corresponding to the first data specifically includes: when continuous sectors are found in the sector corresponding to the first data, the found continuous sectors are merged into one sector, and the one sector and the discontinuous sector in the sector corresponding to the first data are taken as the first abnormal sector.

[0062] It should be noted that finding continuous sectors in the sector corresponding to the first data can be understood as the presence of continuous sectors in the sector corresponding to the first data. Merging the found continuous sectors into one sector can be understood as merging continuous abnormal sectors and treating the merged sector as an abnormal sector. Treating a sector and a discontinuous sector in the sector corresponding to the first data as the first abnormal sector can be understood as treating the merged sector and the discontinuous sector in the sector corresponding to the first data as the first abnormal sector.

[0063] In an embodiment of the present application, the process of determining the first abnormal sector according to the sector corresponding to the first data specifically includes: when no continuous sectors are found in the sector corresponding to the first data, treating the sector corresponding to the first data as the first abnormal sector.

[0064] It should be noted that if no continuous sectors are found in the sectors corresponding to the first data, it can be understood that there are no continuous abnormal sectors in the sectors corresponding to the first data. Taking the sectors corresponding to the first data as the first abnormal sectors can be understood as directly recording the sectors corresponding to the first data as the first abnormal sectors.

[0065] S302: When the capacity information is less than or equal to the first value and / or the quantity information is less than or equal to the second value, shield the first abnormal sector and search for a replica sector of the first abnormal sector in the distributed storage system.

[0066] It should be noted that the first value can be understood as a first ratio of the total capacity information of the first hard disk. In actual applications, the total capacity information of the first hard disk can be determined based on actual conditions and is not limited here. The first ratio can also be determined based on actual conditions and is not limited here. As an example, the first ratio can be 0.1%. The capacity information is less than or equal to the first value, which can be understood as meaning that the total capacity information of the sectors of the first hard disk with hardware damage is less than or equal to 0.1% of the total capacity of the first hard disk.

[0067] It should be noted that the second value can be determined based on actual conditions and is not limited here. As an example, the second value can be 1024. If the number information is less than or equal to the second value, it can be understood that the number of discontinuous damaged sectors in the first hard disk that have hardware damage (i.e., the first abnormal sector) is less than or equal to 1024.

[0068] In an embodiment of the present application, the process of shielding the first abnormal sector specifically includes: obtaining sector list information in the distributed storage system, and shielding the first abnormal sector by removing the first abnormal sector from the sector list information.

[0069] It should be noted that the sector list information can be understood as a free list stored in the distributed storage system.

[0070] The solution of the embodiment of the present application shields the first abnormal sector in the distributed storage system, thereby preventing the second misuse of the first abnormal sector.

[0071] It should be noted that the number of replica sectors of the first abnormal sector in the distributed storage system can be multiple, and the specific number can be determined based on actual conditions and is not limited here. Finding the replica sector of the first abnormal sector in the distributed storage system can be understood as searching the distributed storage system for multiple replica sectors corresponding to the first abnormal sector and selecting any replica sector from the multiple replica sectors.

[0072] S303: Copy the duplicate data in the duplicate sector to other sectors of the first hard disk except the first abnormal sector, so as to access the duplicate data of the first abnormal sector by accessing other sectors.

[0073] It should be noted that the other sectors can be understood as any sector in the first hard disk except the first abnormal sector. The number of other sectors can be one or more, which can be determined according to actual conditions and is not limited here. The copy data can be understood as the data in the copy sector. The data of the copy sector is copied to other sectors in the first hard disk except the first abnormal sector, so that the copy data of the first abnormal sector can be accessed by accessing other sectors. It can be understood that the data of the copy sector is copied to other sectors in the first hard disk except the first abnormal sector, so that the copy data of the first abnormal sector can be accessed by accessing other sectors. It can also be understood that the original data in the first abnormal sector can be accessed by accessing other sectors.

[0074] The solution of the application embodiment can avoid replacing the hard disk and reduce resource waste when a small number of sectors are locally damaged (i.e., the capacity information of the first abnormal sector is less than or equal to the first value and / or the quantity information is less than or equal to the second value), and can quickly repair damaged data with minimal network and processor overhead.

[0075] In an embodiment of the present application, the method further includes: when the capacity information is greater than a first value and / or the quantity information is greater than a second value, outputting indication information, the indication information being used to instruct the hard disk to be replaced.

[0076] It should be noted that the meanings of the first value and the second value are the same as those described above and will not be repeated here. For ease of understanding, an example is given here where, when the total capacity information of the sectors with hardware damage in the first hard disk is greater than 0.1% of the total capacity of the first hard disk, and / or the number of discontinuous damaged sectors of the sectors with hardware damage (i.e., the first abnormal sectors) in the first hard disk is greater than 1024, an indication information is output, and the indication information is used to instruct the hard disk to be replaced. The hard disk replacement process can be understood as replacing the first hard disk with a new one.

[0077] It should be noted that after the first hard disk is replaced with a new hard disk, the data of the new hard disk is reconstructed and restored based on the copy hard disk of the first hard disk in the distributed storage system.

[0078] For ease of understanding, an example is given here to illustrate that in actual applications, the data processing method can be understood as a processing method applied to a distributed storage cluster to prevent hard disk sector damage from affecting business. Figure 4 A schematic diagram of the structure of an exemplary hard disk sector damage optimization solution provided in an embodiment of the present application; Figure 4 As shown, the distributed storage system includes storage server node 1, storage server node 2 and storage server node 3. File A data is stored in three copies on the distributed storage system. On storage server node 1, data A falls to the damaged sector position. The damaged sector is recorded and shielded from the distributed storage system, and the damaged sector data is repaired through the copy.

[0079] Specifically, this system identifies and records damaged hard drive sectors; shields the bad sectors from the distributed storage software system; and, after identifying a hard drive sector failure, repairs the damaged sector data using the distributed storage software system's redundant copies. This eliminates the need to replace a large hard drive after a localized sector failure. Furthermore, redundant copies of data are used to repair the damaged sector data and shield the damaged sector from the system to prevent further misuse.

[0080] The specific steps are as follows:

[0081] (1) Identify hard disk sector damage and issue an alarm;

[0082] (2) shielding damaged sectors from the distributed storage system and counting the capacity of damaged sectors;

[0083] (3) Decide whether to replace the hard disk with a new one based on the capacity of the damaged sectors and repair the damaged data.

[0084] Hard disk sector damage can be discovered in a distributed storage system in two ways: one is triggered by normal business access to the data at the damaged sector location, and the other is the distributed storage system's redundant data consistency scan. When the data at the damaged sector location is accessed, the hard disk damaged sector information will be recorded for subsequent processing. For ease of understanding, Figure 5 A flowchart of an exemplary hard disk sector damage optimization solution identification process provided in an embodiment of the present application; Figure 5 As shown, the steps are as follows:

[0085] 1. Data consistency scan client IO access.

[0086] 2. Hard disk access IO.

[0087] 3. Is it EIO?

[0088] It should be noted that, if EIO is returned, step 4 is executed.

[0089] 4. Determine whether it is a bad sector.

[0090] It should be noted that, if it is determined that the sector is a bad sector (ie, there is a hardware problem in the sector), step 5 is executed.

[0091] 5. Record the damaged sectors of the hard disk.

[0092] 6. Report the record to the management platform.

[0093] Figure 6 A flowchart of an exemplary hard disk sector damage alarm and event recording process provided in an embodiment of the present application; Figure 6As shown, the steps are as follows:

[0094] 1. Management platform.

[0095] 2. Regularly poll to obtain damaged sectors on the hard disk.

[0096] 3. Check the sector capacity.

[0097] 4. Is the capacity greater than 0.1% of the total hard disk capacity or the number of discontinuous damaged sectors greater than 1024?

[0098] It should be noted that, if the capacity is greater than 0.1% of the total hard disk or the number of discontinuous damaged sectors is greater than 1024, execute step 5; if the capacity is less than or equal to 0.1% of the total hard disk or the number of discontinuous damaged sectors is less than or equal to 1024, execute step 6.

[0099] 5. Send hard disk failure alarm and prompt to replace the new disk.

[0100] 6. Report low-level events (hard disk sector damage).

[0101] The management platform periodically polls the hard disk's damaged sector record information. If the total capacity of damaged sectors reaches 0.1% of the hard disk capacity or the number of non-continuous damaged sectors reaches 1024, the hard disk is defined as a faulty disk and an alarm is reported to remind the user to replace the hard disk. If the proportion of damaged sectors to the total hard disk capacity is less than 0.1% and the number of non-continuous damaged sectors is less than 1024, the hard disk is defined as partially damaged, and the customer event information is reported. Hard disks with too many damaged sectors are replaced as faulty disks.

[0102] When the monitoring mechanism finds that a hard disk sector is damaged, it records the information of the damaged hard disk sector. We will block the hard disk sector from the distributed system and decide whether to initiate data repair processing for the damaged sector based on the total capacity of the damaged sectors recorded on the corresponding hard disk. Figure 7 A schematic diagram of an exemplary hard disk sector damage process provided in an embodiment of the present application; Figure 7 As shown, the steps are as follows:

[0103] 1. Storage engine IO access corresponding to the hard disk.

[0104] 2. Return to EIO to determine whether the sector is damaged.

[0105] It should be noted that, if it is determined that the sector is damaged, step 3 is executed.

[0106] 3. The local storage engine records the sector as a damaged sector.

[0107] 4. Whether it can be merged with existing sectors.

[0108] It should be noted that, if there are sectors that can be merged, step 5 is executed; if there are sectors that cannot be merged, step 6 is executed.

[0109] 5. Merge processing.

[0110] It should be noted that step 6 is performed after step 5.

[0111] 6. Record to table.

[0112] 7. Is the capacity greater than 0.1% of the total hard disk capacity or the number of discontinuous damaged sectors greater than 1024?

[0113] It should be noted that, when the capacity is greater than 0.1% of the total hard disk or the number of discontinuous damaged sectors is greater than 1024, execute step 8; when the capacity is less than or equal to 0.1% of the total hard disk or the number of discontinuous damaged sectors is less than or equal to 1024, execute step 11.

[0114] 8. Wait for the faulty disk to be manually replaced.

[0115] 9. After the replacement is completed, the entire disk will be rebuilt and restored.

[0116] 10. End.

[0117] 11. Remove the damaged sector location from the local storage engine freelist.

[0118] 12. Repair damaged sector location data from the copy.

[0119] 13. End.

[0120] The detailed instructions are: Identify damaged hard drive sectors; Merge consecutive damaged sectors and record them; Remove damaged sectors from the local storage engine freelist and calculate the damaged sector capacity and the number of discontinuous damaged sectors; If the damaged sector capacity is too large or the disk is too fragmented, determine the hard drive is faulty and replace it with a new one; If a small number of sectors on the hard drive are damaged, repair the damaged sector data through replication. This process avoids replacing hard drives and wastes resources in the case of localized sector damage, while quickly repairing damaged data with minimal network and processor overhead.

[0121] The present invention provides a data processing device. Figure 8 A structural diagram of a data processing device provided in an embodiment of the present application; Figure 8 As shown, applied to a distributed storage system, the data processing device 800 includes:

[0122] An acquiring unit 801 is configured to, upon detecting that a first abnormal sector occurs on a first hard disk in the distributed storage system, acquire capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector;

[0123] a shielding unit 802 configured to shield the first abnormal sector and search for a replica sector of the first abnormal sector in the distributed storage system when the capacity information is less than or equal to a first value and / or the quantity information is less than or equal to a second value;

[0124] The copy unit 803 is configured to copy the copy data in the copy sector to other sectors of the first hard disk except the first abnormal sector, so as to access the copy data of the first abnormal sector by accessing other sectors.

[0125] Optionally, the data processing device 800 also includes a determination unit, which is used to obtain access data corresponding to the sector of the first hard disk within a first time; when first data representing an access abnormality is found from the access data, determine the sector corresponding to the first data; and determine the first abnormal sector based on the sector corresponding to the first data.

[0126] Optionally, the data processing device 800 also includes a unit for merging the continuous sectors found into one sector when continuous sectors are found in the sector corresponding to the first data, and treating the one sector and the discontinuous sectors in the sector corresponding to the first data as the first abnormal sector.

[0127] Optionally, the as unit is further configured to, when no continuous sectors are found in the sectors corresponding to the first data, regard the sectors corresponding to the first data as the first abnormal sectors.

[0128] Optionally, the data processing device 800 further includes a shielding unit, configured to obtain sector list information in the distributed storage system, and shield the first abnormal sector by removing the first abnormal sector from the sector list information.

[0129] Optionally, the data processing device 800 further includes an output unit, configured to output indication information when the capacity information is greater than the first value and / or the quantity information is greater than the second value, wherein the indication information is used to instruct the hard disk to be replaced.

[0130] The present application also provides a data processing device. Figure 9 A structural diagram of a data processing device provided in an embodiment of the present application; Figure 9As shown, the data processing device 900 includes: a processor 901 and a memory 903 . Optionally, the data processing device 900 may further include a communication bus 902 .

[0131] In a specific embodiment, the processor 901 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor functions may also be other electronic devices, which is not specifically limited in this embodiment.

[0132] In the embodiment of the present application, the communication bus 902 is used to realize the connection and communication between the processor 901 and the memory 903; when the processor 901 executes the running program stored in the memory 903, the following data processing method is implemented:

[0133] When a first abnormal sector is detected in a first hard disk in the distributed storage system, obtaining capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector;

[0134] When the capacity information is less than or equal to a first value, and / or the quantity information is less than or equal to a second value, shielding the first abnormal sector and searching for a replica sector of the first abnormal sector in the distributed storage system;

[0135] The copy data in the copy sector is copied to other sectors of the first hard disk except the first abnormal sector, so that the copy data of the first abnormal sector can be accessed by accessing other sectors.

[0136] Furthermore, the processor 901 is also used to obtain access data corresponding to the sector of the first hard disk within the first time; determine the sector corresponding to the first data when the first data representing the access abnormality is found in the access data; and determine the first abnormal sector based on the sector corresponding to the first data.

[0137] Furthermore, the processor 901 is further configured to, when continuous sectors are found in the sector corresponding to the first data, merge the found continuous sectors into one sector, and use the one sector and the discontinuous sectors in the sector corresponding to the first data as the first abnormal sector.

[0138] Furthermore, the processor 901 is further configured to, when no continuous sectors are found in the sectors corresponding to the first data, treat the sectors corresponding to the first data as the first abnormal sectors.

[0139] Furthermore, the processor 901 is further configured to obtain sector list information in the distributed storage system, and mask the first abnormal sector by removing the first abnormal sector from the sector list information.

[0140] Furthermore, the processor 901 is further configured to output indication information when the capacity information is greater than the first value and / or the quantity information is greater than the second value, wherein the indication information is configured to instruct the hard disk to be replaced.

[0141] An embodiment of the present application provides a storage medium on which a computer program is stored. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors. The computer program implements the data processing method as described above.

[0142] Based on the above embodiments, an embodiment of the present application provides a computer program product, including a computer program, which can be executed by one or more processors, and the computer program implements the data processing method as described above.

[0143] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling an image display device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present disclosure.

[0145] The above description is merely a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: Applied to a distributed storage system, the method includes: When a first abnormal sector is detected in a first hard disk in the distributed storage system, obtaining capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector; When the capacity information is less than or equal to a first value, and / or the quantity information is less than or equal to a second value, shielding the first abnormal sector and searching for a replica sector of the first abnormal sector in the distributed storage system; The copy data in the copy sector is copied to other sectors of the first hard disk except the first abnormal sector, so that the copy data of the first abnormal sector can be accessed by accessing other sectors.

2. The method according to claim 1, characterized in that The detecting that a first abnormal sector appears on a first hard disk in the distributed storage system includes: within a first time, obtaining access data corresponding to a sector of the first hard disk; When first data representing access abnormality is found in the access data, a sector corresponding to the first data is determined; and the first abnormal sector is determined according to the sector corresponding to the first data.

3. The method according to claim 2, characterized in that The determining the first abnormal sector according to the sector corresponding to the first data includes: When continuous sectors are found in the sectors corresponding to the first data, the found continuous sectors are merged into one sector, and the one sector and the discontinuous sectors in the sectors corresponding to the first data are used as the first abnormal sectors.

4. The method according to claim 2, characterized in that The determining the first abnormal sector according to the sector corresponding to the first data includes: When no continuous sectors are found in the sectors corresponding to the first data, the sectors corresponding to the first data are regarded as the first abnormal sectors.

5. The method according to claim 1, wherein The shielding of the first abnormal sector comprises: Sector list information in the distributed storage system is acquired, and the first abnormal sector is shielded by removing the first abnormal sector from the sector list information.

6. The method according to claim 1, characterized in that The method further comprises: In a case where the capacity information is greater than the first value and / or the quantity information is greater than the second value, an indication information is output, where the indication information is used to instruct the hard disk to be replaced.

7. A data processing device, characterized in that: Applied to a distributed storage system, the device includes: an acquiring unit, configured to, upon detecting that a first abnormal sector appears on a first hard disk in the distributed storage system, acquire capacity information of the first abnormal sector and / or information about the number of discontinuous sectors in the first abnormal sector; a shielding unit, configured to shield the first abnormal sector and search for a replica sector of the first abnormal sector in the distributed storage system when the capacity information is less than or equal to a first value and / or the quantity information is less than or equal to a second value; The copy unit is configured to copy the copy data in the copy sector to other sectors of the first hard disk except the first abnormal sector, so as to access the copy data of the first abnormal sector by accessing other sectors.

8. A data processing device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method according to any one of claims 1 to 6 are implemented when the processor executes the program.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images