Storage space management method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a storage space management method and device, a storage medium and electronic equipment, and relates to the field of computers, and the method comprises the steps that to-be-processed object data is acquired; in response to a service request sent by a client, determining a first storage space corresponding to the object data from a storage hard disk of a distributed storage system; when it is detected that the first storage space contains an abnormal storage space, isolating the abnormal storage space, and recording a first space position and first time information of the abnormal storage space; and determining a second storage space required for executing the service request from the storage hard disk, and completing data interaction between the client and the distributed storage system through the second storage space.

Description

Technical Field

[0001] The embodiments of the present application relate to the computer field, and more specifically, to a storage space management method and device, a storage medium, and an electronic device. Background Art

[0002] In a distributed storage system, the hard disk cost of the distributed storage server accounts for an important part of the distributed storage system cost. Bad blocks (marked abnormal storage space) on the hard disk are a common problem in the storage system.

[0003] Among them, the existence of bad blocks may cause data reading and writing failures, affecting the integrity and reliability of data. In the prior art, once a bad block is detected, the silent verification mechanism will release the entire allocated space, causing new write requests to be allocated to the bad block area again, causing write failures. In addition, managers lack effective tools to assess the degree of hard disk damage. Therefore, when a slider is found on the hard disk, a new hard disk may be replaced in time. However, in fact, hard disks with a small number of bad blocks can still be used, which not only wastes space resources, but also increases the cost of using distributed storage, thereby causing the problem of excessively high storage costs of distributed storage systems.

[0004] In the related art, no effective solution has been proposed so far to the problem of excessively high storage costs due to frequent replacement of hard disks in distributed storage systems. Summary of the invention

[0005] The embodiments of the present application provide a storage space management method and device, a storage medium, and an electronic device to at least solve the problem of excessively high storage costs caused by frequent replacement of hard disks in a distributed storage system.

[0006] According to an embodiment of the present application, a storage space management method is provided, including: obtaining object data to be processed; in response to a business request sent by a client, determining a first storage space corresponding to the object data from a storage hard disk of a distributed storage system; when it is detected that the first storage space contains an abnormal storage space, isolating the abnormal storage space and recording a first spatial position and a first time information of the abnormal storage space; determining a second storage space required to execute the business request from the storage hard disk, and completing data interaction between the client and the distributed storage system through the second storage space.

[0007] In an exemplary embodiment, the above-mentioned response to the business request sent by the client determines the first storage space corresponding to the object data from the storage hard disk of the distributed storage system, including: when the business request is to write the object data to the storage hard disk, the first address information of the storage space required to store the object data is allocated by the space allocator, wherein the space allocator is used to count and allocate the storage space of the storage hard disk; based on the first address information, determine the first storage space from the storage hard disk.

[0008] In an exemplary embodiment, after determining the first storage space from the storage hard disk based on the first address information, the method further includes: writing the object data to the first storage space to obtain a first processing result; when the first processing result indicates that data writing fails, detecting the first storage space, and marking and isolating the detected abnormal storage space.

[0009] In an exemplary embodiment, the above-mentioned determining the second storage space required to execute the business request from the storage hard disk and completing the data interaction between the client and the distributed storage system through the second storage space includes: when the business request is to write the object data to the storage hard disk, reallocating the second storage space for the object data through the space allocator, wherein the second storage space does not include the abnormal storage space; and writing the object data to the second storage space.

[0010] In an exemplary embodiment, when the business request is to read the object data from the storage hard disk, the method further includes: in response to the business request sent by the client, reading the object data from the first storage space and returning the reading result; when the reading result indicates a reading failure, detecting the first storage space, determining that the first storage space includes the abnormal storage space, and recording the first space position and the first time information; isolating the abnormal storage space, and releasing the remaining storage space in the first storage space except the abnormal storage space.

[0011] In an exemplary embodiment, after isolating the abnormal storage space and releasing the remaining storage space in the first storage space except the abnormal storage space, the method further includes: reading backup data of the object data from a copy distributed on the target storage node, wherein the backup data contained in the copy is the same as the object data; allocating the second storage space for the backup data through a space allocator, and writing the backup data into the second storage space; and returning the backup data to the client in response to a data read request sent by the client.

[0012] In an exemplary embodiment, the method further includes: determining the sum of abnormal storage spaces of a group of abnormal storage spaces recorded in a historical period based on the first spatial location information; determining to replace the storage hard disk with a target storage hard disk when the ratio between the sum of the abnormal storage spaces and the total storage space of the storage hard disk is greater than or equal to a preset threshold; or determining a group of time information when the group of abnormal storage spaces is detected in the historical period based on the first time information; determining the occurrence frequency of the abnormal storage space based on the group of time information; and determining to replace the storage hard disk with the target storage hard disk when the occurrence frequency satisfies a preset growth rule, wherein, The preset growth rule includes that within adjacent time intervals, the number of times the abnormal storage space appears in the time interval after the time point is greater than the number of times the abnormal storage space appears in the time interval before the time point; or based on the first spatial position information, determining the sum of the abnormal storage spaces of a group of abnormal storage spaces recorded in the historical period; determining the ratio between the sum of the abnormal storage spaces and the total storage space of the storage hard disk; based on the set of time information, determining the frequency of occurrence of the abnormal storage space; when the ratio is greater than or equal to the preset threshold and the frequency of occurrence of the abnormal storage space satisfies the preset growth rule, determining to replace the storage hard disk with the target storage hard disk.

[0013] According to another embodiment of the embodiment of the present application, a storage space management device is also provided, including: an acquisition unit, used to acquire object data to be processed; a first processing unit, used to determine a first storage space corresponding to the object data from a storage hard disk of a distributed storage system in response to a business request sent by a client; a second processing unit, used to isolate the abnormal storage space when it is detected that the first storage space contains an abnormal storage space, and record the first spatial position and first time information of the abnormal storage space; a third processing unit, used to determine a second storage space required to execute the business request from the storage hard disk, and complete the data interaction between the client and the distributed storage system through the second storage space.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] According to another embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0017] Through the above-mentioned embodiments provided by the present application, when executing a business request sent by a client on the hard disk of a distributed storage server, for example, when performing a read or write operation on the hard disk, if the read or write operation fails, the hard disk is detected and the detected abnormal storage space is isolated. Then, the processing flow corresponding to the business request is completed using the reallocated second storage space, thereby avoiding the object data from being written to the marked abnormal storage space again. At the same time, the utilization rate of the storage space is effectively improved, the probability of replacing the hard disk is reduced, and the problem of excessively high storage costs caused by frequent replacement of hard disks in the distributed storage system is solved, thereby achieving the technical effect of reducing the storage cost of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a server device of a storage space management method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of an optional storage space management method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional OSD read and write service process function module according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of processing bad block space of an optional write service according to an embodiment of the present application;

[0023] Figure 5is a flowchart of processing bad block space of an optional read service according to an embodiment of the present application;

[0024] Figure 6 is a structural block diagram of a verification device for a soft decoding function according to an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] The storage space management method embodiment provided in the embodiment of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a storage space management method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the storage space management method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In order to better understand the above-mentioned storage space management method, the basic concepts of the professional terms and terminology used in the embodiments of the present application are briefly introduced below.

[0032] OSD: Object-based Storage Device, also known as storage node, is mainly responsible for storing and retrieving data. A node has multiple OSD processes, each OSD is a process that mounts a data hard disk, and the process can read and write data on the hard disk;

[0033] Monitor (MON): The main task is to maintain the consistency of the cluster view. When maintaining consistency, the Paxos algorithm (protocol) is used and instantiated into the database for subsequent access. In the storage architecture, Monitor is one of the key management nodes. They are the core components of the cluster, monitoring the health status of all node osd services, monitoring metadata and configuration, and ensuring the consistency of various parts of the cluster;

[0034] OSDMap: used to describe the status and topology of OSDs in the cluster, including OSD status and health information, PG distribution information, CRUSH Map version number, and other metadata related to cluster topology, configuration, and status, so that CephMonitor can monitor and manage the entire storage cluster in real time;

[0035] Storage pool: A storage pool is a logical partition in a distributed storage system and is a collection of PGs. It is divided into two types: replica pool and erasure pool. A replica pool means that the object data between PG members is the same;

[0036] PG: Placement Group, a carrier for placing objects. Multiple PGs are created in a storage pool;

[0037] Object: Object is the smallest unit of data storage, which is distributed in each PG. A PG stores a large number of objects.

[0038] Missing table: In distributed storage, the Missing table usually refers to the list of data objects that are lost or missing in the object storage. Through the object recovery process, the object data of this replica can be read from other replicas for recovery to achieve data redundancy.

[0039] In order to solve the above problems existing in the related art, a storage space management method is provided in this embodiment, and its execution subject is not limited to a distributed storage server, wherein the distributed storage server includes but is not limited to two parts: hardware and software. The hardware includes but is not limited to a hard disk of the distributed storage server, and the software includes but is not limited to an execution logic for executing data reading and writing services, an execution logic for space management, etc. Figure 2 As shown, the process includes the following steps S202-S208:

[0040] Step S202: Obtain object data to be processed;

[0041] Step S204: In response to the service request sent by the client, determine a first storage space corresponding to the object data from the storage hard disk of the distributed storage system;

[0042] Step S206: when it is detected that the first storage space includes an abnormal storage space, isolating the abnormal storage space, and recording a first spatial position and a first time information of the abnormal storage space;

[0043] Step S208: determining a second storage space required for executing the service request from the storage hard disk, and completing data interaction between the client and the distributed storage system through the second storage space.

[0044] Before explaining the above storage space management method, we first Figure 3 The following is a brief introduction to the OSD read and write business process function modules.

[0045] like Figure 3 As shown, a new OSD read-write business process function module proposed in the technical solution of this application is a part of a distributed storage server, wherein the functions of each part in the function module are as follows:

[0046] 1) The space allocator is responsible for applying for and releasing the space on the hard disk for object data in the read and write business;

[0047] 2) The free space manager is responsible for recording the location information of all free blocks on the hard disk;

[0048] When an OSD is created for the first time, the free space manager will initialize the free location information based on the hard disk size and the hard disk minimum allocation unit block size, and record the initialized information in the key-value database. During the subsequent OSD service restart process, the free space manager will read the free location information from the key-value database for initialization.

[0049] Since the free position information provided by the free space manager can be used to reversely calculate the used position information, the space allocator then manages the space of the entire hard disk and provides interfaces for applying for and releasing space for read and write services. Each application or release of the space allocator will update the free position of the free space manager and update the key-value database in real time.

[0050] 3) The bad block check module is used to check the physical control applied for the object data in the case of read and write failure, and filter out the bad block space, and release the space that is not a bad block through the space allocator release interface;

[0051] 4) The bad block list management module is used to record the bad block information encountered during the reading and writing process, and save the bad block information in the KeyValueDB database, where the bad block information includes the location of the bad block and the bad block time.

[0052] In addition, the bad block list management module also provides bad block sector mapping statistics, which can count the number of bad blocks contained in each sector of the hard disk and the number of newly added bad blocks according to the time axis, and present a bad block statistical analysis chart on the management interface for managers to analyze the health of the hard disk and determine whether the hard disk needs to be replaced. The following will describe in detail the implementation method of determining whether a new hard disk needs to be replaced in conjunction with a specific embodiment.

[0053] The above-mentioned business requests include but are not limited to read data requests and write data requests sent by the client to the distributed storage server, among which the read data request may be but is not limited to reading data from the storage space in the hard disk of the distributed storage server and returning the read data to the client; the write data request may be but is not limited to writing the data sent by the client to a preset space location in the hard disk of the distributed storage server.

[0054] In a distributed storage system, the abnormal storage space of the hard disk can be understood as the "bad block space" of the hard disk, or the bad block of the hard disk for short. The bad block space refers to the space on the hard disk that cannot store or read data normally due to physical damage or other reasons. The storage of bad block space may affect the reliability of the distributed storage system and the integrity of the data.

[0055] The above-mentioned response to the business request sent by the client determines the first storage space corresponding to the object data from the storage hard disk of the distributed storage system. It can be understood, but not limited to, that when the business request is to write the object data to the hard disk, the storage space is applied for for it according to the data size of the object data through the space allocator; when the business request is to read the object data from the hard disk, the storage space at the current spatial position of the object data is queried from the storage space of the hard disk.

[0056] When it is detected that the first storage space includes an abnormal storage space, the abnormal storage space is isolated. For example, when the service request is to write object data to a hard disk, if the writing fails, the abnormal storage space is isolated by using Figure 3 The bad block check module shown checks the physical space applied for the object data, screens out the bad block space, and isolates the bad block space, so that when there is a new write request, new object data is prevented from being written again into the detected bad block space.

[0057] At the same time, the space that is not a bad block is released through the space allocator release interface, and the bad block space information is saved through the bad block record interface of the bad block list management module.

[0058] The above-mentioned determination of the second storage space required to execute the business request from the storage hard disk means that when the business request is to write the object data to the hard disk, if the writing fails, new storage space is re-applied for the object data through the space allocator until the object data is successfully written to the hard disk.

[0059] In the same way, when the business request is to read object data from the hard disk, the first storage space is the storage space at the spatial location where the object data is currently located, and the second storage space is the storage space of the object data in other copies when the object data is successfully read from the spatial location where the object data is stored in other copies in the event of a read failure.

[0060] In a distributed storage system, replicas can refer to, but are not limited to, multiple copies of data, which are distributed on different storage nodes to improve data availability, reliability, and fault tolerance. The replica mechanism is one of the key technologies to ensure data security and system stability in a distributed storage system.

[0061] Common types of replicas include synchronous replicas, asynchronous replicas, strong consistency replicas, and eventual consistency replicas. In practical applications, the replica strategy needs to make a trade-off between data security, storage cost, system performance, and complexity. The more replicas there are, the higher the data security and availability, but the storage cost and system complexity also increase. Therefore, the designer of the distributed storage system needs to choose an appropriate replica strategy based on specific business requirements and resource conditions.

[0062] By adopting the above method, when executing a business request sent by a client on the hard disk of a distributed storage server, for example, when performing a read or write operation on the hard disk, if the read or write operation fails, the hard disk is detected and the detected abnormal storage space is isolated. Then, the processing flow corresponding to the business request is completed using the reallocated second storage space, thereby avoiding writing the object data to the marked abnormal storage space again. At the same time, the utilization rate of the storage space is effectively improved, the probability of replacing the hard disk is reduced, and the problem of excessively high storage costs caused by frequent replacement of hard disks in the distributed storage system is solved, thereby achieving the technical effect of reducing the storage cost of the distributed storage system.

[0063] In an exemplary embodiment, in response to the service request sent by the client, determining the first storage space corresponding to the object data from the storage hard disk of the distributed storage system includes:

[0064] In the case where the service request is to write the object data into the storage hard disk, first address information of the storage space required to store the object data is allocated by a space allocator, wherein the space allocator is used to count and allocate the storage space of the storage hard disk;

[0065] Based on the first address information, the first storage space is determined from the storage hard disk.

[0066] When the service request is to write the object data to the hard disk, it can be done by, but not limited to, Figure 4 This is achieved by the processing flow of bad blocks for write services shown in FIG.

[0067] S402, applying for storage space for the object data through a space allocator;

[0068] When the client triggers the object data writing business process, the space allocator will allocate continuous hard disk space for the new object data as much as possible and return the requested space address to the client.

[0069] S404, writing the object data into the applied hard disk space;

[0070] According to the space address returned in step S402, the object data is written into the storage space corresponding to the space address, and the writing result is returned.

[0071] S406, determining whether the writing is successful;

[0072] If the writing fails, execute step S408; otherwise, execute step S414.

[0073] S408, checking the allocated hard disk space (which can also be understood as storage space) through a bad block checking module;

[0074] If it is detected that there is abnormal storage space (ie bad block space) in the currently applied storage space, the bad block space is marked.

[0075] It should be noted that, assuming that the storage space required to store the object data is 4k, in the event of a write failure, if it is detected that 1k of the 4k space is a bad block space, the 1k storage space will be determined as the bad block space, and the remaining 3k space will still be available normal storage space.

[0076] S4 1 0, release the non-bad block space through the release interface of the space allocator;

[0077] The bad block space is the 1k space in the 4k space in the above embodiment, and the non-bad block space is the remaining 3k space.

[0078] S412, saving the spatial location of the bad block and the time information of the current bad block through the bad block list management module;

[0079] S4 1 4, update the spatial location information in the object metadata.

[0080] It can be seen that through the above space management method in the case of write failure, when the write fails and the bad block space is detected, the bad block space can be managed without releasing the operation, so when the new object data is written to the hard disk, the space allocator will not apply for the free space of the bad block again. This will not only prevent the newly written object data from being reallocated to the released bad block space control, but also extend the service life of the hard disk.

[0081] It should be noted that the above-mentioned method of detecting whether there is a bad block space handles the above-mentioned case of a write failure and synchronously starts the detection process. In addition, it can also periodically detect whether there is a bad block space in the hard disk space at a preset time interval, so that when writing object data, the space allocator will not apply for free space marked as a bad block space for the object data to be written. That is, invalid writing to the bad block space is avoided from the root, and the data writing efficiency is improved.

[0082] In an exemplary embodiment, after determining the first storage space from the storage hard disk based on the first address information, the method further includes:

[0083] Writing the object data into the first storage space to obtain a first processing result;

[0084] When the first processing result indicates that data writing fails, the first storage space is detected, and the detected abnormal storage space is marked and isolated.

[0085] In the process of writing the object data into the storage space applied for by the space allocator, a first processing result is obtained, and the first processing result is returned to the client in real time.

[0086] The first processing result is used to indicate whether the data is written successfully or failed. When the first processing result indicates that the data is written failed, the first storage space is detected, and the detected abnormal storage space is marked and isolated. At the same time, the bad block list management module also records the bad block information detected during the writing process and saves the bad block information in the key-value database.

[0087] Through the above marking, when new object data needs to be written into the hard disk, the space allocator avoids allocating the marked bad block free space to the new object data according to the marking, thereby reducing invalid writing of the object data.

[0088] In an exemplary embodiment, the above-mentioned determining the second storage space required for executing the service request from the storage hard disk and completing the data interaction between the client and the distributed storage system through the second storage space includes:

[0089] In the case where the service request is to write the object data into the storage hard disk, reallocating the second storage space for the object data by a space allocator, wherein the second storage space does not include the abnormal storage space;

[0090] The object data is written into the second storage space.

[0091] In combination with the description in the above embodiments, it can be seen that when writing fails, it will be detected whether the first storage space applied for the object data by the space allocator includes abnormal storage space, and if it is detected that it contains abnormal storage space, the abnormal storage space will be marked, and the remaining storage space in the first storage space except the abnormal storage space will be released.

[0092] In addition, the space allocator will reapply for new available storage space for the object data, so as to write the object data into the reapplied second storage space.

[0093] Obviously, the second storage space may include the unmarked normal storage space in the first storage space (for example, the intact 3k space in the above embodiment), or may include part of the unmarked normal storage space in the first storage space, or may not include the unmarked normal storage space in the first storage space.

[0094] Through the above method, when bad block space is identified in the writing process, the bad block space can be isolated and not released to ensure that the object data is written again to the storage space containing the bad block space, thereby ensuring the reliability of data writing and maximizing the use of the available storage space in the hard disk, thereby improving the utilization rate of the hard disk space.

[0095] In addition to the above business requests for writing data, there are also business requests for reading target object data from the hard disk space. For read requests, they specifically include:

[0096] In response to the service request sent by the client, read the object data from the first storage space and return the reading result;

[0097] In the case where the reading result indicates a reading failure, detecting the first storage space, determining that the first storage space includes the abnormal storage space, and recording the first space position and the first time information;

[0098] The abnormal storage space is isolated, and remaining storage space in the first storage space except the abnormal storage space is released.

[0099] Through Figure 5 The flowchart shown implements the management of storage space.

[0100] S502, reading object data from the hard disk according to the spatial location information recorded in the object metadata;

[0101] After the client triggers the object data read business process, the metadata information of the object is obtained and the storage location information of the object on the hard disk is parsed.

[0102] Metadata is data used to describe object data, which is crucial for data management, retrieval, and operation. Metadata includes but is not limited to file and directory attributes, location information, access permissions, etc. In a distributed file system, metadata operations account for a large part of all file system operations, and effective metadata performance is crucial to overall system performance.

[0103] The role of metadata in distributed storage systems includes:

[0104] 1) File directory tree maintenance: Metadata maintains the tree structure of the file system and records the hierarchical relationship between files and directories;

[0105] 2)2) Data block management: Each file is divided into multiple data blocks. The metadata records the detailed information of these data blocks, including the unique identifier, size, number of backups, and storage location of the data blocks;

[0106] 3) Storage node information: The metadata also contains information about all storage nodes (slave nodes), such as the number of nodes, the IP address of each node, disk usage, etc.

[0107] 4) Performance optimization: Through metadata, the system can optimize data storage and access. For example, through inode prefetching technology, read I / O interactions with storage devices can be reduced, improving performance.

[0108] Through the above metadata information, the spatial location information (spatial address and storage time, etc.) used to store the object data when storing the object data can be obtained.

[0109] S504, determining whether the reading fails;

[0110] If it fails, execute step S506; otherwise, end the process.

[0111] Among them, according to the obtained hard disk space location information, the object data is read from the hard disk. If the read is successful, the read object data is returned to the client, and the object process ends.

[0112] S506, if the read fails, the hard disk space allocated to the object is checked by a bad block check module to separate the bad block space from the non-bad block space;

[0113] Isolate the separated bad block space, that is, do not release the bad block space, and perform the following operations at the same time:

[0114] Reading backup data of the object data from a copy distributed on a target storage node, wherein the backup data contained in the copy is the same as the object data;

[0115] Allocate the second storage space for the backup data by a space allocator, and write the backup data into the second storage space;

[0116] In response to the data read request sent by the client, the backup data is returned to the client.

[0117] Please refer to the following steps S508 to S514 for details.

[0118] S508, releasing the control information without bad blocks through a release interface of the hard disk space allocator;

[0119] S5 1 0, the bad block list management module saves the spatial location of the bad block and the time information of the current bad block.

[0120] S 51 2, put the object data into the missing table, and recover the object through the data recovery process;

[0121] The object recovery process may be, but is not limited to, reading backup data of the object data from a replica. The meaning and function of the replica may refer to the description in the above embodiment, which will not be repeated here.

[0122] It should be noted that in the object recovery process, after obtaining the object data read from the replica, the hard disk space is repartitioned for the object data through the space allocator, and the object data read from the replica is written to the new hard disk space, and the spatial location information of the object metadata information is updated.

[0123] Common types of replicas include:

[0124] 1) Synchronous replicas: When a write operation occurs, the data is written to all replicas. The write operation is considered complete only when all replicas have been successfully written;

[0125] 2) Asynchronous replicas: Data is first written to one replica and then asynchronously replicated to other replicas. This approach can improve write performance but may increase the risk of data loss in the short term;

[0126] 3) Strong consistency replicas: ensure that all replicas are consistent at all times, that is, read operations can always obtain the latest data;

[0127] 4) Eventually consistent replicas: There is no guarantee that all replicas will be consistent immediately, but after a period of time, all replicas will eventually reach a consistent state.

[0128] The number and distribution strategy of replicas depends on the design and requirements of the distributed storage system, for example:

[0129] 1) Three-copy strategy: Three copies of data are stored on different nodes, which can tolerate two nodes failing at the same time without losing data;

[0130] 2) Multi-copy strategy: More copies can be set as needed to provide higher data reliability.

[0131] Replicas also involve data reconstruction and migration. When a storage node fails, the system will start the data reconstruction process and rebuild the lost replicas from other replicas. In addition, when the system is expanded or load balanced, replicas may be migrated between nodes to optimize the use of storage resources.

[0132] In practical applications, the replication strategy needs to make a trade-off between data security, storage cost, system performance, and complexity. The more replicas there are, the higher the data security and availability, but the storage cost and system complexity also increase. Therefore, the designer of the distributed storage system needs to choose an appropriate replication strategy based on specific business requirements and resource conditions.

[0133] S514, after the object recovery is completed, the object data reading business process is re-triggered, and the read object data is returned to the client.

[0134] It should be noted that the above step S512 is an object recovery process of the distributed storage system, which reads the object data from the replica to complete the recovery of the object data of the replica, including the update of the spatial location information of the object data.

[0135] Through the embodiments of the present application, the bad block location information of the hard disk can be effectively managed and isolated through the bad block list management module, and the use time of the hard disk can be extended without affecting the read and write services, avoiding the situation where the hard disk needs to be replaced when a bad block appears. This not only extends the service life of the distributed storage, but also saves the cost of frequent hard disk replacement.

[0136] In an exemplary embodiment, the management of the storage space further includes:

[0137] Based on the first spatial position information, determining the sum of abnormal storage spaces of a group of abnormal storage spaces recorded in a historical period;

[0138] When the ratio of the sum of the abnormal storage space to the total storage space of the storage hard disk is greater than or equal to a preset threshold, determining to replace the storage hard disk with a target storage hard disk; or

[0139] Based on the first time information, determining a set of time information at which the set of abnormal storage spaces is detected within the historical period;

[0140] Based on the set of time information, determining the occurrence frequency of the abnormal storage space;

[0141] In the case where the occurrence frequency satisfies a preset growth rule, determining to replace the storage hard disk with the target storage hard disk, wherein the preset growth rule includes that within adjacent time intervals, the number of times the abnormal storage space appears in the time interval at a later time point is greater than the number of times the abnormal storage space appears in the time interval at an earlier time point; or

[0142] Based on the first spatial position information, determining the sum of abnormal storage spaces of a group of abnormal storage spaces recorded in a historical period;

[0143] Determine the ratio between the sum of the abnormal storage space and the total storage space of the storage hard disk;

[0144] Based on the set of time information, determining the occurrence frequency of the abnormal storage space;

[0145] When the proportion is greater than or equal to the preset threshold and the occurrence frequency of the abnormal storage space meets the preset growth rule, it is determined to replace the storage hard disk with the target storage hard disk.

[0146] Combined with the description in the above embodiment, it can be known that the bad block list management module can be used to record the bad block information encountered during the reading and writing process, and save the bad block information in the key-value database. In addition, the bad block list management module also provides a bad block sector mapping statistics function, which can count the number of bad blocks contained in each sector of the hard disk and the number of newly added bad blocks according to the time axis, and present a bad block statistical analysis chart on the management interface, so that the management personnel can analyze the health of the hard disk and determine whether the hard disk needs to be replaced.

[0147] For example, when a write operation fails, the recorded bad block space location information and time information are counted into the historical data in the historical period. For example, the total number of bad blocks is increased by 1, or the total space occupied by bad blocks is increased.

[0148] In a specific embodiment, assuming that the ratio of the sum of the storage space of the bad block space counted in the past year to the total hard disk space is greater than a preset threshold, the administrator will consider replacing a new hard disk to ensure the reliability of data storage in the distributed storage system. In other words, ensure that data can be accurately read and written in the distributed storage system.

[0149] In another optional embodiment, it is also possible to determine whether a new hard disk needs to be replaced based on the frequency of bad blocks. For example, assuming that the preset time interval is 1 month, the number of bad blocks in the first month is 5 times, the number of bad blocks in the second month is 10, and the number of bad blocks in the third month is 30 times. This pattern of step-like growth or explosive growth can determine that the hard disk space loss is relatively serious. The later the time point, the higher the frequency of bad blocks. Therefore, a comprehensive assessment of whether a new hard disk needs to be replaced can be made based on the service life and usage time of the hard disk.

[0150] In another optional embodiment, it is also possible to determine whether a new hard disk needs to be replaced by combining the frequency of bad block occurrence and the ratio of the sum of bad block space to the total hard disk space. Specifically, when the ratio of the sum of bad block space to the total hard disk space is greater than a preset threshold and the frequency of bad block occurrence meets a preset growth rule, a new hard disk is replaced.

[0151] Through the above management strategies, the hard disk design and use of the distributed storage system can maximize the use of normal space, while reducing the impact of bad block space on system performance and data integrity. Through software-level optimization and hardware-level selection, the bad block problem of the hard disk can be accepted to a certain extent, while also improving the reliability of data storage services.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0153] In this embodiment, a storage space management device is also provided to implement the above-mentioned embodiments and preferred implementation modes, which have been described and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the modules described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0154] Figure 6 : is a structural block diagram of a storage space management device according to an embodiment of the present application, the device comprising:

[0155] An acquisition unit 602 is used to acquire object data to be processed;

[0156] A first processing unit 604 is configured to determine, in response to a service request sent by a client, a first storage space corresponding to the object data from a storage hard disk of the distributed storage system;

[0157] The second processing unit 606 is configured to isolate the abnormal storage space when detecting that the first storage space includes an abnormal storage space, and record a first spatial position and first time information of the abnormal storage space;

[0158] The third processing unit 608 is used to determine the second storage space required to execute the service request from the storage hard disk, and complete the data interaction between the client and the distributed storage system through the second storage space.

[0159] In an exemplary embodiment, the above-mentioned first processing unit 604 includes: a first allocation module, which is used to allocate the first address information of the storage space required to store the object data through a space allocator when the business request is to write the object data into the storage hard disk, wherein the space allocator is used to count and allocate the storage space of the storage hard disk; a first processing module, which is used to determine the first storage space from the storage hard disk based on the first address information.

[0160] In an exemplary embodiment, the above-mentioned first processing module includes: a writing submodule, used to write the object data into the first storage space to obtain a first processing result; a first detection submodule, used to detect the first storage space when the first processing result indicates that data writing fails, and mark and isolate the detected abnormal storage space.

[0161] In an exemplary embodiment, the third processing unit 608 includes: a second allocation module, used to reallocate the second storage space for the object data through a space allocator when the business request is to write the object data into the storage hard disk, wherein the second storage space does not include the abnormal storage space; and a writing module, used to write the object data into the second storage space.

[0162] In an exemplary embodiment, the above-mentioned device also includes: a reading unit, which is used to read the object data from the first storage space in response to the business request sent by the client, and return the reading result; a detection unit, which is used to detect the first storage space when the reading result indicates a reading failure, determine that the first storage space contains the abnormal storage space, and record the first space position and the first time information; an isolation unit, which is used to isolate the abnormal storage space and release the remaining storage space in the first storage space except the abnormal storage space.

[0163] In an exemplary embodiment, the above-mentioned device also includes: a reading module, which is used to read the backup data of the object data from the copy distributed on the target storage node after isolating the abnormal storage space and releasing the remaining storage space in the first storage space except the abnormal storage space, wherein the backup data contained in the copy is the same as the object data; a third allocation module, which is used to allocate the second storage space to the backup data through a space allocator and write the backup data into the second storage space; and a second processing module, which is used to return the backup data to the client in response to a data read request sent by the client.

[0164] In an exemplary embodiment, the above-mentioned device also includes: a fourth processing unit, which is used to determine the sum of abnormal storage spaces of a group of abnormal storage spaces recorded in a historical period based on the first spatial position information; when the ratio between the sum of the abnormal storage spaces and the total storage space of the storage hard disk is greater than or equal to a preset threshold, determine to replace the storage hard disk with a target storage hard disk; or based on the first time information, determine a group of time information of the group of abnormal storage spaces detected in the historical period; based on the group of time information, determine the frequency of occurrence of the abnormal storage space; when the frequency of occurrence meets a preset growth law, determine to replace the storage hard disk with the target storage hard disk. , wherein the preset growth rule includes that within adjacent time intervals, the number of times the abnormal storage space appears in the time interval after the time point is greater than the number of times the abnormal storage space appears in the time interval before the time point; or based on the first spatial position information, determining the sum of the abnormal storage spaces of a group of abnormal storage spaces recorded in the historical period; determining the ratio between the sum of the abnormal storage spaces and the total storage space of the storage hard disk; based on the set of time information, determining the frequency of occurrence of the abnormal storage space; when the ratio is greater than or equal to the preset threshold and the frequency of occurrence of the abnormal storage space satisfies the preset growth rule, determining to replace the storage hard disk with the target storage hard disk.

[0165] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0166] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0167] Optionally, in this embodiment, the computer program may be configured to execute the following steps through the computer program: S1, obtaining object data to be processed;

[0168] S2, in response to a service request sent by the client, determining a first storage space corresponding to the object data from a storage hard disk of the distributed storage system;

[0169] S3, when it is detected that the first storage space includes an abnormal storage space, isolating the abnormal storage space and recording a first spatial position and a first time information of the abnormal storage space;

[0170] S4, determining a second storage space required to execute the service request from the storage hard disk, and completing data interaction between the client and the distributed storage system through the second storage space.

[0171] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0172] The embodiment of the present application also provides an electronic device, such as Figure 7 As shown, the electronic device includes a memory 702 and a processor 704. The memory 702 stores a computer program, and the processor 704 is configured to execute the steps in any of the above method embodiments through the computer program.

[0173] Optionally, in this embodiment, the processor 704 may be configured to perform the following steps through a computer program:

[0174] S1, obtaining the object data to be processed;

[0175] S2, in response to a service request sent by the client, determining a first storage space corresponding to the object data from a storage hard disk of the distributed storage system;

[0176] S3, when it is detected that the first storage space includes an abnormal storage space, isolating the abnormal storage space and recording a first spatial position and a first time information of the abnormal storage space;

[0177] S4, determining a second storage space required to execute the service request from the storage hard disk, and completing data interaction between the client and the distributed storage system through the second storage space.

[0178] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0179] Alternatively, a person skilled in the art may understand that: Figure 7 The structure shown is for illustration only. Figure 7 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 7 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 7 Different configurations are shown.

[0180] Among them, the memory 702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the storage space management method and the alarm information processing device in the embodiment of the present application. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, realizing the above-mentioned storage space management method. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 702 may further include a memory remotely located relative to the processor 704, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 702 can be specifically, but not limited to, used to store object data, first spatial location information, and first time information, etc. As an example, such as Figure 7 As shown, the memory 702 may include, but is not limited to, the acquisition unit 602, the first processing unit 604, the second processing unit 606, and the third processing unit 608 in the storage space management device. In addition, other module units in the verification device of the soft decoding function may also be included but are not limited to the above, which will not be repeated in this example.

[0181] Optionally, the transmission device 706 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 706 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0182] In addition, the electronic device further includes: a display 708; and a connection bus 710, which is used to connect various module components in the electronic device.

[0183] In other embodiments, the electronic device may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0184] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0185] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0186] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0187] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0188] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A storage space management method, characterized in that: include: Get the object data to be processed; In response to a service request sent by a client, determining a first storage space corresponding to the object data from a storage hard disk of the distributed storage system; In the case where it is detected that the first storage space includes an abnormal storage space, isolating the abnormal storage space and recording a first spatial position and a first time information of the abnormal storage space; A second storage space required for executing the service request is determined from the storage hard disk, and data interaction between the client and the distributed storage system is completed through the second storage space.

2. The method according to claim 1, characterized in that The step of determining, in response to the service request sent by the client, a first storage space corresponding to the object data from a storage hard disk of the distributed storage system includes: In the case where the service request is to write the object data into the storage hard disk, first address information of the storage space required to store the object data is allocated by a space allocator, wherein the space allocator is used to count and allocate the storage space of the storage hard disk; Based on the first address information, the first storage space is determined from the storage hard disk.

3. The method according to claim 2, characterized in that After determining the first storage space from the storage hard disk based on the first address information, the method further includes: Writing the object data into the first storage space to obtain a first processing result; When the first processing result indicates that data writing fails, the first storage space is detected, and the detected abnormal storage space is marked and isolated.

4. The method according to claim 1, characterized in that The determining, from the storage hard disk, a second storage space required for executing the service request, and completing data interaction between the client and the distributed storage system through the second storage space includes: In the case where the service request is to write the object data into the storage hard disk, reallocating the second storage space for the object data by a space allocator, wherein the second storage space does not include the abnormal storage space; The object data is written into the second storage space.

5. The method according to claim 1, characterized in that In the case where the service request is to read the object data from the storage hard disk, the method further includes: In response to the service request sent by the client, read the object data from the first storage space and return the reading result; When the reading result indicates a reading failure, the first storage space is detected to determine that the first storage space includes the abnormal storage space, and the first space position and the first time information are recorded; the abnormal storage space is isolated, and the remaining storage space in the first storage space except the abnormal storage space is released.

6. The method according to claim 5, characterized in that After isolating the abnormal storage space and releasing the remaining storage space in the first storage space except the abnormal storage space, the method further includes: Reading backup data of the object data from a copy distributed on a target storage node, wherein the backup data contained in the copy is the same as the object data; Allocate the second storage space for the backup data by a space allocator, and write the backup data into the second storage space; In response to the data read request sent by the client, the backup data is returned to the client.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Based on the first spatial position information, determining the sum of abnormal storage spaces of a group of abnormal storage spaces recorded in a historical period; When the ratio of the sum of the abnormal storage space to the total storage space of the storage hard disk is greater than or equal to a preset threshold, determining to replace the storage hard disk with a target storage hard disk; or Based on the first time information, determining a set of time information at which the set of abnormal storage spaces is detected within the historical period; Based on the set of time information, determining the occurrence frequency of the abnormal storage space; In the case where the occurrence frequency satisfies a preset growth rule, determining to replace the storage hard disk with the target storage hard disk, wherein the preset growth rule includes that within adjacent time intervals, the number of times the abnormal storage space appears in the time interval at a later time point is greater than the number of times the abnormal storage space appears in the time interval at an earlier time point; or Based on the first spatial position information, determining the sum of abnormal storage spaces of a group of abnormal storage spaces recorded in a historical period; Determine the ratio between the sum of the abnormal storage space and the total storage space of the storage hard disk; Based on the set of time information, determining the occurrence frequency of the abnormal storage space; When the proportion is greater than or equal to the preset threshold and the occurrence frequency of the abnormal storage space meets the preset growth rule, it is determined to replace the storage hard disk with the target storage hard disk.

8. A storage space management device, characterized in that: include: An acquisition unit, used for acquiring object data to be processed; A first processing unit, configured to determine, in response to a service request sent by a client, a first storage space corresponding to the object data from a storage hard disk of the distributed storage system; a second processing unit, configured to isolate the abnormal storage space when detecting that the first storage space includes an abnormal storage space, and record a first spatial position and first time information of the abnormal storage space; The third processing unit is used to determine the second storage space required to execute the business request from the storage hard disk, and complete the data interaction between the client and the distributed storage system through the second storage space.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor. When executed, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images