Cluster Information Scanning Method and Apparatus, Electronic Device, and Storage Medium

Abstract

The present application provides a method and apparatus for scanning cluster information, an electronic device, and a storage medium. The method includes: determining the corresponding relationship between each of a plurality of sub-threads and a storage unit in the cluster, where the cluster includes a plurality of storage units, and each sub-thread corresponds to at least one storage unit; according to the corresponding relationship, concurrently executing the plurality of sub-threads to obtain storage unit information corresponding to each storage unit, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information; and merging all the storage unit information to obtain the cluster information of the cluster. By using the method in this embodiment, by processing the storage units in the cluster with a plurality of sub-threads, the storage unit information of multiple storage units can be determined simultaneously, effectively overcoming the problem of low efficiency in scanning information in the cluster in the related art.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular, to a method and device for scanning cluster information, an electronic device, and a storage medium. Background Art

[0002] The existing distributed cloud storage service can solve relevant complex problems such as storage expansion, data reliability and security, and distributed access for users.

[0003] In order to ensure the security and reliability of cluster data, it is necessary to regularly scan all information in the cluster corresponding to the distributed cloud storage service. Due to the huge amount of data in the cluster, it becomes very time-consuming to scan this information in the cluster. For a cluster at the level of tens of millions of storage units, the time consumption usually exceeds 3 minutes, so the time consumption is relatively long, which causes troubles for the maintenance and security of the cluster, and is not conducive to online operation and maintenance.

[0004] As can be seen from the above, there is a problem of low efficiency in scanning information in the cluster in the related technology. Summary of the Invention

[0005] This application provides a method and device for scanning cluster information, an electronic device, and a storage medium, so as to at least solve the problem of low efficiency in scanning information in the cluster in the related technology.

[0006] According to one aspect of the embodiments of the present application, a method for scanning cluster information is provided, including:

[0007] Determine the corresponding relationship between each of the multiple sub-threads and the storage units in the cluster, where the cluster includes multiple storage units, and each sub-thread corresponds to at least one storage unit;

[0008] According to the corresponding relationship, by concurrently executing the multiple sub-threads, obtain the storage unit information corresponding to each storage unit, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information;

[0009] Merge all the storage unit information to obtain the cluster information of the cluster.

[0010] Optionally, as the foregoing method, the merging all the storage unit information to obtain the cluster information of the cluster includes:

[0011] Determine the storage unit information obtained by each sub-thread;

[0012] In the main thread, merge all the storage unit information to obtain the cluster information.

[0013] Optionally, as in the foregoing method, after merging all the storage unit information to obtain the cluster information of the cluster, the method further includes:

[0014] Total all the storage unit information to obtain the data volume corresponding to each storage unit type;

[0015] Based on the data volume corresponding to each storage unit type and the preset storage data volume in the cluster, obtain determination information for indicating whether the scanning of the cluster is completed.

[0016] Optionally, as in the foregoing method, the obtaining the storage unit information corresponding to each storage unit by concurrently executing the multiple sub-threads according to the corresponding relationship includes:

[0017] According to the corresponding relationship, determine all the target storage units corresponding to the sub-thread;

[0018] Through the sub-thread, query orphan blocks, triple replicas, and erasure code replicas in all the target storage units, where each triple replica has a corresponding erasure code replica, and the orphan block is an erasure code replica that does not have a corresponding triple replica.

[0019] Optionally, as in the foregoing method, the querying for orphan blocks in all the target storage units by the sub-thread includes:

[0020] Determine a triple replica set and an erasure code replica set, where the triple replica set includes all the triple replicas, and the erasure code replica set includes all the erasure code replicas;

[0021] Based on the preset corresponding relationship between the triple replicas and the erasure code replicas, determine a target erasure code replica set corresponding to the triple replica set, where any erasure code replica in the target erasure code replica set has a corresponding triple replica;

[0022] By taking the difference set of the target erasure code replica set and the erasure code replica set, determine the orphan blocks in the erasure code replica set, where the orphan blocks do not exist in the target erasure code replica set and exist in the erasure code replica set.

[0023] Optionally, as in the foregoing method, after querying for orphan blocks in all the target storage units by the sub-thread, the method further includes:

[0024] Determine the first total number of all the orphan blocks;

[0025] When the ratio of the first total number to the second total number of all the erasure copies is greater than or equal to a preset ratio, determine the health of the cluster as a sub-healthy state.

[0026] Optionally, as in the foregoing method, after merging all the storage unit information to obtain the cluster information of the cluster, the method further includes:

[0027] Determine specified three-replica sets from all the three-replica sets, where the specified three-replica sets are three-replica sets in which the current number of other three-replica sets identical to the specified three-replica sets among all the three-replica sets is less than a preset number;

[0028] Determine the missing number based on the difference between the preset number and the current number;

[0029] Copy the specified three-replica sets to generate the missing number of newly generated three-replica sets.

[0030] According to another aspect of the embodiments of the present application, there is also provided a cluster information scanning device, including:

[0031] A determination module, configured to determine the correspondence between each of a plurality of child threads and storage units in a cluster, where the cluster includes a plurality of the storage units, and each of the child threads corresponds to at least one of the storage units;

[0032] An acquisition module, configured to obtain the storage unit information corresponding to each storage unit by concurrently executing the plurality of child threads according to the correspondence, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information;

[0033] A merging module, configured to merge all the storage unit information to obtain the cluster information of the cluster.

[0034] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory communicate with each other through the communication bus; where the memory is used to store a computer program; and the processor is configured to execute the method steps in any of the foregoing embodiments by running the computer program stored on the memory.

[0035] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, where a computer program is stored in the storage medium, and the computer program is configured to execute the method steps in any of the foregoing embodiments when running.

[0036] In the embodiments of the present application, a correspondence between each of a plurality of child threads and storage units in a cluster is determined, where the cluster includes a plurality of the storage units, and each of the child threads corresponds to at least one of the storage units; according to the correspondence, by concurrently executing the plurality of child threads, storage unit information corresponding to each of the storage units is obtained, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information; by merging all the storage unit information, the cluster information of the cluster is obtained. By means of the method in this embodiment, by using a plurality of child threads to process the storage units in the cluster, the storage unit information of a plurality of storage units can be determined simultaneously, and by merging all the storage unit information, the cluster information corresponding to the cluster can be obtained. Therefore, the efficiency of obtaining the cluster information of the cluster can be effectively improved, and the problem of low information scanning efficiency in the cluster in the related art can be effectively overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 is a schematic flowchart of an optional method for scanning cluster information according to an embodiment of the present application;

[0040] Figure 2 is a schematic flowchart of an optional method for scanning cluster information according to another embodiment of the present application;

[0041] Figure 3 is a schematic diagram of an optional method for scanning cluster information according to another embodiment of the present application;

[0042] Figure 4 is a block diagram of the structure of an optional device for scanning cluster information according to an embodiment of the present application;

[0043] Figure 5 is a block diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solution in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0045] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0046] According to one aspect of the embodiments of this application, a method for scanning cluster information is provided. Optionally, in this embodiment, the above method for scanning cluster information can be applied to a hardware environment composed of servers. Moreover, the server is connected to the terminal through a network and can be used to provide services (such as data storage services, cloud computing services, etc.) for the terminal or the client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0047] The above network can include but is not limited to at least one of the following: wired network, wireless network. The above wired network can include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network. The above wireless network can include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal is not limited to being a PC, mobile phone, tablet computer, etc.

[0048] The method for scanning cluster information in the embodiments of this application can be executed by the server, or can be executed by the terminal, or can be jointly executed by the server and the terminal. Among them, when the terminal executes the method for scanning cluster information in the embodiments of this application, it can also be executed by the client installed on it.

[0049] Taking the execution of the method for scanning cluster information in this embodiment by the server as an example, as Figure 1 shown, an optional method for scanning cluster information provided by the embodiments of this application includes the following steps:

[0050] Step S101: Determine the correspondence between each of multiple sub-threads and the storage units in the cluster. Here, the cluster includes multiple storage units, and each sub-thread corresponds to at least one storage unit.

[0051] The cluster information scanning method in this embodiment can be applied to scenarios such as determining whether the data in the cluster is secure and reliable and whether there is data loss. For example, identifying whether there is a scenario of data replica loss in a cluster providing cloud storage services, identifying whether there is a scenario of data replica loss in a cluster providing distributed cloud storage services, etc., or it can also be identifying other scenarios of data loss. In the embodiments of the present application, the above-mentioned cluster information scanning method is illustrated by taking the scenario of identifying whether there is a data replica loss in a cluster providing distributed cloud storage services as an example. For other types of scenarios, the above-mentioned cluster information scanning method is equally applicable when there is no contradiction.

[0052] Taking the scenario of identifying whether there is a data replica loss in a cluster providing distributed cloud storage services as an example, all the data in all clusters is scanned to determine whether there is any missing data in the cluster.

[0053] A trigger instruction for triggering the execution of the above step S101 can be obtained in advance. For example, a scan instruction for instructing to scan the cluster is obtained. Then, step S101 can be: According to the scan instruction, determine the correspondence between each of the multiple sub-threads and the storage units in the cluster.

[0054] In the related art, a single thread is used to traverse all the Blocks (i.e., storage units) in the cluster, so there is a problem of low efficiency.

[0055] In this embodiment, by setting multiple sub-threads and determining the storage units corresponding to each sub-thread. Optionally, the number of sub-threads can be determined based on the number of storage units in the cluster. For example, it is preset that each sub-thread corresponds to N storage units (N is an integer greater than or equal to 1, such as 10000, etc.). Furthermore, based on the total number M of storage units in the cluster and N, the number P of sub-threads can be determined. If M is an integer multiple of N, then P = M / N; if M is not an integer multiple of N, then P = [M / N] + 1. Further, the number of all storage units can be evenly distributed to each sub-thread, that is, the number Q of storage units allocated to each sub-thread = M / P.

[0056] Each storage unit has a corresponding Block ID. Therefore, by establishing the correspondence between the child threads and the Block IDs, the correspondence between each child thread and the storage unit can be determined. Subsequently, each child thread only needs to scan the storage unit indicated by its corresponding Block ID in the later stage.

[0057] Step S102, according to the correspondence, by concurrently executing multiple child threads, obtain the storage unit information corresponding to each storage unit, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information.

[0058] After determining the correspondence between the child threads and the storage units, all the child threads can be concurrently executed according to this correspondence, and each child thread processes its corresponding storage unit to obtain the storage unit information of the storage unit.

[0059] The storage unit information can be information that can indicate the type and data volume of the storage unit corresponding to the storage unit information. For example: the type of the storage unit can be triple replication or erasure code replication, and the data volume can be 1MB or 2MB, etc.

[0060] For example, when there are child threads I, II, and III, and child thread I corresponds to storage units 0 - 9999, child thread II corresponds to storage units 10000 - 19999, and child thread III corresponds to storage units 20000 - 29999, the following operations are performed simultaneously: child thread I processes storage units 0 - 9999 in sequence to obtain the storage unit information of each storage unit in storage units 0 - 9999; child thread II processes storage units 10000 - 19999 to obtain the storage unit information of each storage unit in storage units 10000 - 19999; child thread III processes storage units 20000 - 29999 to obtain the storage unit information of each storage unit in storage units 20000 - 29999.

[0061] Step S103, merge all the storage unit information to obtain the cluster information of the cluster.

[0062] After obtaining all the storage unit information, the cluster information of the cluster can be obtained by merging all the storage unit information.

[0063] As can be seen from the foregoing, the storage unit information includes the type and data volume of the storage unit. Therefore, the cluster information can include the type and data volume of each storage unit, and the total data volume of each type of storage unit.

[0064] Through the method in this embodiment, by using multiple child threads to process the storage units in the cluster, the storage unit information of multiple storage units can be determined simultaneously, and by merging all the storage unit information, the cluster information corresponding to the cluster can be obtained. Therefore, the efficiency of obtaining the cluster information of the cluster can be effectively improved, and the problem of low information scanning efficiency in the cluster existing in the related art can be effectively overcome.

[0065] As Figure 2 shown, as an optional implementation manner, for the method as described above, step S103 of merging all the storage unit information to obtain the cluster information of the cluster includes the following steps:

[0066] Step S201, determine the storage unit information obtained by each child thread;

[0067] Step S202, in the main thread, merge all the storage unit information to obtain the cluster information.

[0068] After executing step S102, the storage unit information of the storage unit corresponding to each child thread has been obtained through each child thread in the multiple child threads.

[0069] All child threads can summarize the processed storage unit information to the main thread. Optionally, a child thread can transmit it to the main thread immediately after obtaining each storage unit information, or after processing all the storage unit information corresponding to it, then transmit all the storage unit information to the main thread.

[0070] When the main thread obtains the storage unit information, it can classify each storage unit information according to each child thread to determine all the storage unit information obtained by each child thread in the main thread, and determine whether each child thread has completed scanning all the corresponding storage units.

[0071] After the main thread obtains all the storage unit information, the main thread can total all the storage unit information to determine the total data volume in the cluster and the sub - total data volume of each type of data based on the type and data volume of the storage unit indicated by each storage unit information. For example: when the types of storage units include triple - copy type and erasure - code copy type, the total data volume can be the sum of the data volumes of the storage units of the triple - copy type and the storage units of the erasure - code copy type, and the sub - total data volume can include: the sum of the data volumes of all the storage units of the triple - copy type, and / or the sum of the data volumes of all the storage units of the erasure - code copy type.

[0072] Through the method in this embodiment, an implementation manner is provided in which, by setting a main thread, on the basis of obtaining the storage unit information through child threads, the cluster information of the cluster can still be obtained.

[0073] As an alternative implementation, in the method as described above, after merging all the storage unit information in step S103 to obtain the cluster information of the cluster, the method further includes the following steps:

[0074] Step S301, total all the storage unit information to obtain the data volume corresponding to each storage unit type.

[0075] After obtaining all the storage unit information, statistics can be performed according to the storage unit types in the storage unit information to obtain the data volume corresponding to each storage unit type.

[0076] Step S302, based on the data volume corresponding to each storage unit type and the preset storage data volume in the cluster, obtain judgment information for indicating whether the scanning of the cluster is completed.

[0077] After obtaining the data volume corresponding to each storage unit type, judgment information can be obtained based on the preset storage data volume in the cluster.

[0078] The preset storage data volume can be the data volume of the original data to be stored.

[0079] The judgment information can be information for indicating whether the data in the cluster has been completely scanned.

[0080] After obtaining the data volume corresponding to each storage unit type and the preset storage data volume in the cluster, judgment information corresponding to the cluster can be obtained based on the above information.

[0081] For example, when storing the original data with triple replicas, the sum of the data volumes corresponding to the triple replica type (i.e., the total amount of sub-data of the triple replica type) should be 3 times the preset storage data volume. Therefore, when the total amount of sub-data of the triple replica type is 3 times the preset storage data volume, the judgment information indicates that the cluster data is secure and reliable and there is no missing data replica; when the total amount of sub-data of the triple replica type is less than 3 times the preset storage data volume, the judgment information indicates that the cluster data is unreliable and there is a missing data replica.

[0082] Through the method in this embodiment, judgment information for indicating whether the data in the cluster is reliable can be directly generated, so that the cluster data can be maintained in a timely manner, the reliability of the cluster data can be improved, and the integrity of relevant data in the later stage can be avoided from being affected.

[0083] As an alternative implementation, in the method as described above, step S102 obtains the storage unit information corresponding to each storage unit by concurrently executing multiple sub-threads according to the corresponding relationship, including the following steps:

[0084] Step S401: Determine all target storage units corresponding to the child thread according to the corresponding relationship.

[0085] After determining the corresponding relationship, for any one of all child threads, all target storage units corresponding to the child thread can be determined.

[0086] For example, in the case where the corresponding relationship is established by establishing the corresponding relationship between the child thread and the Block ID (i.e., the ID of the storage unit), all target storage units corresponding to the child thread can be determined through the target Block ID corresponding to the child thread.

[0087] Step S402: Query for orphan blocks, triple replicas, and erasure-coded replicas in all target storage units through the child thread, where each triple replica has a corresponding erasure-coded replica, and an orphan block is an erasure-coded replica without a corresponding triple replica.

[0088] After determining all target storage units corresponding to the child thread, the child thread can be used to query for orphan blocks, triple replicas, and erasure-coded replicas in all corresponding target storage units.

[0089] Generally: To improve data reliability and security, the cluster stores data in three copies, namely triple replicas, one primary replica and two secondary replicas. For example, they can be distinguished by 0, 1, 2. And generally, the same three triple replicas are stored in different nodes respectively to avoid the loss of all the same triple replicas in case one node fails; Erasure-coded replicas: The triple replicas use EC coding (i.e., erasure coding, a coding method that saves data storage space) to store data on the erasure-coded replicas, thereby saving data storage space. Each erasure-coded replica has a unique Block ID. Generally, different erasure-coded replicas corresponding to the same triple replica are stored in different nodes respectively; And, normally, the Block of a triple replica uniquely maps to the Block of an erasure-coded replica, and after each triple replica is encoded by EC, four corresponding erasure-coded replicas can be obtained, and each erasure-coded replica is stored in a different node.

[0090] Through the method in this embodiment, the child thread can query for replicas in its corresponding target storage units and determine the specific types of the replicas (i.e., orphan blocks, triple replicas, and erasure-coded replicas), which is convenient for later judging whether there are missing situations for various types of replicas based on the types of the replicas.

[0091] As Figure 3 shown, as an optional implementation manner, for the method as described above, step S402 of querying for orphan blocks in all target storage units through the child thread includes the following steps:

[0092] Step S501: Determine the triple-copy set and the erasure-copy set. Among them, the triple-copy set includes all triple copies, and the erasure-copy set includes all erasure copies.

[0093] After determining all triple copies and all erasure copies, the triple-copy set including all triple copies and the erasure-copy set including all erasure copies can be determined.

[0094] Step S502: Based on the preset correspondence between triple copies and erasure copies, determine the target erasure-copy set corresponding to the triple-copy set. Among them, any erasure copy in the target erasure-copy set has a corresponding triple copy.

[0095] As known from the foregoing embodiments, since erasure copies are obtained by performing EC encoding on triple copies, in normal cases, each triple copy in the cluster should have a corresponding erasure copy.

[0096] For example, for any triple copy, after obtaining the erasure copy corresponding to the triple copy, the preset correspondence can be obtained by associating the Block ID of the storage unit where the erasure copy is stored with the Block ID corresponding to the triple copy.

[0097] After determining the triple-copy set, the erasure copies corresponding to each triple copy in the triple-copy set can be determined according to the preset correspondence, and the target erasure-copy set including all erasure copies corresponding to triple copies can be obtained.

[0098] Step S503: By taking the difference set of the target erasure-copy set and the erasure-copy set, determine the orphan blocks in the erasure-copy set. Among them, the orphan blocks do not exist in the target erasure-copy set but exist in the erasure-copy set.

[0099] After obtaining the target erasure-copy set and the erasure-copy set, the difference set calculation can be performed on the target erasure-copy set and the erasure-copy set, and the erasure copies that exist in the erasure-copy set but do not exist in the target erasure-copy set can be determined as orphan blocks.

[0100] Since the erasure copies in the target erasure-copy set have corresponding triple copies, the orphan blocks are the erasure copies that do not have corresponding triple copies. Therefore, the erasure copies corresponding to the lost triple copies and the lost triple copies in the cluster can be determined.

[0101] Through the method in this embodiment, orphan blocks can be determined among all erasure copies, so that the lost triple copies in the cluster can be determined, which is convenient for timely maintenance of the data in the cluster, thereby improving the integrity of the cluster data.

[0102] As an alternative implementation, in the method as described above, after querying for orphan blocks in all target storage units through a sub-thread in step S402, the method further includes the following steps:

[0103] Step S601, determining a first total number of all orphan blocks.

[0104] After determining all orphan blocks, the orphan blocks can be counted to determine the first total number of all orphan blocks.

[0105] Step S602, when the ratio of the first total number to a second total number of all erasure copies is greater than or equal to a preset ratio, determining the health of the cluster as a sub-healthy state.

[0106] After determining all erasure copies, the second total number of erasure copies can be determined, and then the ratio between the first total number and the second total number can be calculated to obtain the ratio of the first total number to the second total number of all erasure copies.

[0107] After obtaining the ratio, by comparing the ratio with the preset ratio, and when the ratio is greater than or equal to the preset ratio, determining the health of the cluster as a sub-healthy state.

[0108] The preset ratio can be pre-set and is used to indicate the minimum ratio of orphan blocks for the cluster to be in a sub-healthy state.

[0109] For example, when the first total number is 1000, the second total number is 10000000, and the preset ratio is 0.001%, then the ratio of the first total number to the second total number is 0.01%, which is greater than the preset ratio. Therefore, the health of the cluster is in a sub-healthy state.

[0110] Through the method in this embodiment, the health of the cluster can be determined in real time to facilitate real-time maintenance of the data in the cluster, thereby improving the reliability and security of the cluster.

[0111] As an alternative implementation, in the method as described above, after merging all storage unit information to obtain the cluster information of the cluster in step S103, the method further includes the following steps:

[0112] Step S701, determining a specified triple copy among all triple copies, where the specified triple copy is a triple copy in which the current number of other triple copies identical to the specified triple copy among all triple copies is less than a preset number;

[0113] Step S702, determining a missing number based on the difference between the preset number and the current number;

[0114] Step S703: Copy the specified triple-copy to obtain the missing number of newly generated triple-copies.

[0115] After obtaining all the storage unit information, all the triple-copies in the cluster can be determined, and the specified triple-copy can be determined among all the triple-copies.

[0116] Under normal circumstances, the same copy can store 3 identical triple-copies in the cluster; therefore, for any one of the triple-copies, there are 2 other triple-copies that are the same as this triple-copy, that is, the preset number is 2. When the current number of other triple-copies that are the same as the specified triple-copy is only 1 or 0, it means that there is a loss of other triple-copies that are the same as this specified triple-copy. Therefore, the missing number can be determined based on the difference between the current number and the preset number.

[0117] After determining the missing number, the specified triple-copy can be copied to obtain the newly generated triple-copies with the missing number to supplement the lost triple-copies.

[0118] Through the method in this embodiment, any specified triple-copy with a missing copy in the cluster can be supplemented to ensure that the number of triple-copies in the cluster remains normal, thereby improving the security and integrity of the cluster data.

[0119] As described below, an application example applying any of the foregoing embodiments is provided:

[0120] 1) Change the process of traversing all Blocks (i.e., storage units) in the cluster in a single-threaded manner before to a multi-threaded concurrent processing. Each thread only processes the corresponding Block and obtains the Block information (i.e., storage unit information) of the corresponding Block, including the type of these Blocks and the corresponding data volume, and returns them separately. Then, merge the Block information in all the child threads in the main thread.

[0121] 2) For each thread, optimize the method of finding orphan Blocks. In the process of finding orphan Blocks (i.e., orphan chunks), for the erasure code chunks without corresponding triple-copies, they are called orphan chunks. In the process of finding orphan chunks, the method of directly finding the difference set is adopted.

[0122] According to another aspect of the embodiments of the present application, a cluster information scanning device is further provided, including:

[0123] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

[0125] According to another aspect of the embodiments of this application, there is also provided a cluster information scanning device for implementing the above cluster information scanning method. The structural block diagram of an optional cluster information scanning device according to the embodiments of this application is as Figure 4 shown. The device may include:

[0126] A determination module 1, configured to determine the correspondence between each of a plurality of child threads and storage units in a cluster, where the cluster includes a plurality of storage units, and each child thread corresponds to at least one storage unit;

[0127] An acquisition module 2, configured to obtain storage unit information corresponding to each storage unit by concurrently executing a plurality of child threads according to the correspondence, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information;

[0128] A merging module 3, configured to merge all the storage unit information to obtain the cluster information of the cluster.

[0129] It should be noted that the determination module in this embodiment can be used to execute the above step S101, the acquisition module 2 in this embodiment can be used to execute the above step S102, and the merging module 3 in this embodiment can be used to execute the above step S103.

[0130] Specifically, for the specific process of each module in the device according to the embodiments of the present invention to implement its functions, reference may be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0131] As an optional embodiment, for the aforementioned cluster information scanning device, the merging module 3 is used for:

[0132] Determine the storage unit information obtained by each sub-thread;

[0133] In the main thread, merge all the storage unit information to obtain the cluster information.

[0134] Specifically, for the specific process of each module in the device according to the embodiments of the present invention to implement its functions, reference may be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0135] As an optional embodiment, it further includes a judgment module, which is used for:

[0136] Total all the storage unit information to obtain the data volume corresponding to each storage unit type;

[0137] Based on the data volume corresponding to each storage unit type and the preset storage data volume in the cluster, obtain judgment information for indicating whether the scanning of the cluster is completed.

[0138] Specifically, for the specific process of each module in the device according to the embodiments of the present invention to implement its functions, reference may be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0139] As an optional embodiment, for the aforementioned cluster information scanning device, the obtaining module 2 is used for:

[0140] Determine the sub-module, which is used to determine all target storage units corresponding to the sub-thread according to the corresponding relationship;

[0141] The query sub-module is used to query orphan blocks, triple replicas, and erasure code replicas through the sub-thread in all target storage units, where each triple replica has a corresponding erasure code replica, and the orphan block is an erasure code replica without a corresponding triple replica.

[0142] Specifically, for the specific process of each module in the device according to the embodiments of the present invention to implement its functions, reference may be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0143] As an optional embodiment, for the aforementioned cluster information scanning device, the query sub-module is used for:

[0144] Determine the triple replica set and the erasure code replica set, where the triple replica set includes all triple replicas, and the erasure code replica set includes all erasure code replicas;

[0145] Determine a target erasure copy set corresponding to the triple copy set based on a preset correspondence between the triple copies and the erasure copies, where any erasure copy in the target erasure copy set has a corresponding triple copy.

[0146] By taking the difference set of the target erasure copy set and the erasure copy set, orphan blocks are determined in the erasure copy set, where the orphan blocks do not exist in the target erasure copy set and exist in the erasure copy set.

[0147] Specifically, for the specific processes of each module in the device according to the embodiments of the present invention to implement their functions, reference can be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0148] As an alternative embodiment, the cluster information scanning device as described above further includes a health degree judgment module for:

[0149] Determine the first total number of all orphan blocks;

[0150] When the ratio of the first total number to the second total number of all erasure copies is greater than or equal to a preset ratio, determine the health degree of the cluster as a sub-healthy state.

[0151] Specifically, for the specific processes of each module in the device according to the embodiments of the present invention to implement their functions, reference can be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0152] As an alternative embodiment, the cluster information scanning device as described above further includes a replication module for:

[0153] Determine a specified triple copy among all triple copies, where the specified triple copy is a triple copy in all triple copies for which the current number of other triple copies identical to the specified triple copy is less than a preset number;

[0154] Determine the missing number based on the difference between the preset number and the current number;

[0155] Replicate the specified triple copy to obtain a new number of newly generated triple copies equal to the missing number.

[0156] Specifically, for the specific processes of each module in the device according to the embodiments of the present invention to implement their functions, reference can be made to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0157] It should be noted here that the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run in the hardware environment for implementing the method as shown in Figure 1 and can be implemented by software or by hardware, where the hardware environment includes a network environment.

[0158] According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the above-mentioned cluster information scanning method, and the electronic device may be a server, a terminal, or a combination thereof.

[0159] According to another embodiment of the present application, there is also provided an electronic device, including: as Figure 5 shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504. Among them, the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0160] The memory 1503 is used to store a computer program;

[0161] When the processor 1501 is used to execute the program stored on the memory 1503, the following steps are implemented:

[0162] Step S101: Determine the correspondence between each sub-thread in multiple sub-threads and the storage units in the cluster. Among them, the cluster includes multiple storage units, and each sub-thread corresponds to at least one storage unit.

[0163] Step S102: According to the correspondence, by concurrently executing multiple sub-threads, obtain the storage unit information corresponding to each storage unit, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information.

[0164] Step S103: Merge all the storage unit information to obtain the cluster information of the cluster.

[0165] Optionally, in this embodiment, the above-mentioned communication bus may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0166] The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0167] The above-mentioned processor can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit, central processing unit), NP (Network Processor, network processor), etc.; it can also be a DSP (Digital Signal Processor, digital signal processor), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), FPGA (Field-Programmable Gate Array, field-programmable gate array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0168] The embodiments of the present application further provide a computer-readable storage medium. The storage medium includes a stored program. When the program runs, it executes the method steps of the above-mentioned method embodiments.

[0169] Optionally, in this embodiment, the above-mentioned storage medium may include but not limited to: various media that can store program codes such as USB flash drives, ROMs, RAMs, mobile hard disks, magnetic disks, or optical discs.

[0170] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments.

[0171] If the integrated unit in the above-mentioned embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the above-mentioned computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in the storage medium and includes several instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0172] In the above-mentioned embodiments of the present application, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] In several embodiments provided by this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0174] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0175] In addition, in each embodiment of this application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0176] The above is only the preferred embodiment of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A method for scanning cluster information, characterized in that, Including: Determine the corresponding relationship between each of the multiple child threads and the storage units in the cluster, where the cluster includes multiple storage units, and each child thread corresponds to at least one storage unit; According to the corresponding relationship, by concurrently executing the multiple child threads, obtain the storage unit information corresponding to each storage unit, including: according to the corresponding relationship, determine all target storage units corresponding to the child thread; query orphan blocks, triple replicas, and erasure code replicas in all the target storage units through the child thread, where the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information, each triple replica has a corresponding erasure code replica, and the orphan block is an erasure code replica that does not have a corresponding triple replica; Merge all the storage unit information to obtain the cluster information of the cluster.

2. The method according to claim 1, wherein The merging all the storage unit information to obtain the cluster information of the cluster includes: Determine the storage unit information obtained by each child thread; In the main thread, merge all the storage unit information to obtain the cluster information.

3. The method according to claim 1, characterized in that After the merging all the storage unit information to obtain the cluster information of the cluster, the method further includes: Total all the storage unit information to obtain the data volume corresponding to each storage unit type; Based on the data volume corresponding to each storage unit type and the preset storage data volume in the cluster, obtain judgment information for indicating whether the scanning of the cluster is completed.

4. The method according to claim 1, characterized in that The querying orphan blocks in all the target storage units through the child thread includes: Determine a triple replica set and an erasure code replica set, where the triple replica set includes all the triple replicas, and the erasure code replica set includes all the erasure code replicas; Based on the preset corresponding relationship between the triple replicas and the erasure code replicas, determine a target erasure code replica set corresponding to the triple replica set, where any erasure code replica in the target erasure code replica set has a corresponding triple replica; By taking the difference set of the target erasure code replica set and the erasure code replica set, determine the orphan blocks in the erasure code replica set, where the orphan blocks do not exist in the target erasure code replica set and exist in the erasure code replica set.

5. The method according to claim 1, wherein After the querying orphan blocks in all the target storage units through the child thread, the method further includes: Determine the first total number of all the orphan blocks; In the case where the proportion of the first total number in the second total number of all the erasure code replicas is greater than or equal to a preset proportion, determine the health status of the cluster as a sub-healthy state.

6. The method according to any one of claims 1 to 5, characterized in that, After the merging all the storage unit information to obtain the cluster information of the cluster, the method further includes: Determine a specified triple replica among all the triple replicas, where the specified triple replica is a triple replica in all the triple replicas whose current number of other triple replicas identical to the specified triple replica is less than a preset number; Determine the missing quantity based on the difference between the preset quantity and the current quantity; Copy the specified triple copy to obtain a new generated triple copy of the missing quantity.

7. A cluster information scanning device, characterized in that, Comprising: A determination module, configured to determine the correspondence between each of the multiple sub-threads and the storage units in the cluster, wherein the cluster includes a plurality of the storage units, and each of the sub-threads corresponds to at least one of the storage units; An acquisition module, configured to obtain the storage unit information corresponding to each of the storage units by concurrently executing the multiple sub-threads according to the correspondence, including: determining all target storage units corresponding to the sub-thread according to the correspondence; querying orphan blocks, triple copies, and erasure code copies in all the target storage units through the sub-thread, wherein the storage unit information is used to indicate the type and data volume of the storage unit corresponding to the storage unit information, each triple copy has a corresponding erasure code copy, and the orphan block is an erasure code copy without a corresponding triple copy; A merging module, configured to merge all the storage unit information to obtain the cluster information of the cluster.

8. An electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory complete communication with each other through the communication bus, characterized in that The memory is used to store a computer program; The processor is configured to execute the method steps described in any one of claims 1 to 6 by running the computer program stored on the memory.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the storage medium, wherein the computer program is set to execute the method steps described in any one of claims 1 to 6 when running.

Images