Data processing method and device, equipment and storage medium

Abstract

The application relates to the computer technical field and discloses a data processing method, device and equipment and a storage medium, which are applied to an unstructured storage system and comprise the following steps: obtaining a to-be-written file and an initial storage pool list, the initial storage pool list storing a plurality of normal storage pools isolated according to storage nodes; for any normal storage pool, recognizing a storage pool capacity of the normal storage pool, and determining a reference storage pool and a capacity proximity difference based on the storage pool capacity of the normal storage pool, wherein the capacity proximity difference is used for representing that the storage pool capacities of the normal storage pools are in the same data capacity proximity difference; based on the reference storage pool, screening the normal storage pools in the initial storage pool list which are in the capacity proximity difference to obtain a target storage list; and based on file information of the to-be-written file, determining a target storage pool for storing the to-be-written file from the target storage list; the application can reduce the internal overhead of a single storage pool and reduce the influence of a storage pool fault on a service.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] In distributed unstructured storage, especially in big data distributed file systems with separate storage and compute, a storage pool is typically used to store data. However, in large-scale distributed unstructured storage systems, a single storage pool usually manages multiple disks, leading to excessive internal links, severe internal resource consumption, and even system failure. Furthermore, because the storage pool is closely related to the fault domain, a problem with the storage pool will affect the input and output of the entire storage service. Summary of the Invention

[0003] In view of this, the present invention provides a data processing method, apparatus, device and storage medium to solve the problem in existing distributed unstructured storage that, since a storage pool usually manages multiple disks, there are too many internal links in the storage pool, resulting in serious internal consumption, and when the storage pool has a problem, it will affect the business input and output of the entire namespace.

[0004] Firstly, this invention provides a data processing method applied to an unstructured storage system. The method includes: acquiring a file to be written and an initial storage pool list, wherein the initial storage pool list stores multiple storage nodes, each storage node stores at least one normal storage pool, and the normal storage pools are isolated according to storage nodes; for any normal storage pool, identifying the storage pool capacity of the normal storage pool, and determining a baseline storage pool and a capacity similarity difference based on the storage pool capacity of the normal storage pool, the capacity similarity difference being used to characterize the similarity difference between the storage pool capacities of various normal storage pools at the same data capacity; based on the baseline storage pool, filtering the normal storage pools in the initial storage pool list that are within the capacity similarity difference to obtain a target storage list; and based on the file information of the file to be written, determining the target storage pool for storing the file to be written from the target storage list. Through the above process, the amount of disk data within the storage pool can be reduced, allowing for decentralized management and ensuring system stability; simultaneously, after the storage pools are grouped according to nodes, new fault domains are formed, and single-node failures are handled and recovered within the storage pool without affecting other storage pools, effectively isolating faulty nodes and reducing the impact of storage pool failures on business operations.

[0005] In some optional implementations, based on the file information of the file to be written, a target storage pool for storing the file to be written is determined from a target storage list, including:

[0006] Get the file information of the file to be written, including the file name or file path;

[0007] Perform a hash operation on the file information to obtain a hash value;

[0008] The hash value is moduloed based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written.

[0009] In some optional implementations, based on a baseline storage pool, normal storage pools within a similar capacity range in the initial storage pool list are filtered to obtain a target storage list, including:

[0010] Get the storage pool capacity of each normal storage pool in the initial storage pool list;

[0011] Calculate the capacity difference between the normal storage pool capacity and the baseline storage pool capacity;

[0012] The capacity difference is compared with the capacity similarity difference, and the normal storage pools in the initial storage pool list are filtered based on the comparison results to obtain the target storage list.

[0013] In some optional implementations, the normal storage pools in the initial storage pool list are filtered based on the comparison results to obtain a target storage list, including:

[0014] When the comparison results indicate that the capacity difference is less than or equal to the capacity similarity difference, the normal storage pool will be filtered out from the initial storage pool list;

[0015] The target storage list is obtained based on the selected normal storage pools.

[0016] In some alternative implementations, determining a baseline storage pool and capacity similarity differences based on the storage pool capacity of a normal storage pool includes:

[0017] Retrieve the normal storage pool with the smallest capacity from the initial storage pool list;

[0018] The normal storage pool with the smallest storage pool capacity is determined as the baseline storage pool;

[0019] Based on the capacity differences of each normal storage pool, the capacity similarity difference is determined.

[0020] In some optional implementations, obtaining an initial list of storage pools includes:

[0021] Obtain the storage node to which each data disk belongs;

[0022] Based on the storage nodes, corresponding storage pools are divided for the data disks;

[0023] Determine the original list of storage pools based on the storage pools corresponding to the storage nodes;

[0024] The initial storage pool list is determined based on the storage status of each storage pool in the original storage pool list.

[0025] In some alternative implementations, the initial storage pool list is determined based on the storage status of each storage pool in the original storage pool list;

[0026] Monitor the service status of the storage pool to obtain the storage pool status;

[0027] When the storage pool status indicates that the service status of the storage pool is abnormal, the storage pools with abnormal service status in the original storage pool list are removed to obtain the initial storage pool list.

[0028] Secondly, the present invention provides a data processing device applied to an unstructured storage system. The device mainly includes: an information acquisition module, an information determination module, a list determination module, and a file storage module. The information acquisition module acquires the file to be written and an initial storage pool list. The initial storage pool list stores multiple storage nodes, and each storage node stores at least one normal storage pool, with the normal storage pools isolated according to the storage nodes. The information determination module identifies the storage pool capacity of any normal storage pool and determines a benchmark storage pool and a capacity similarity difference based on the normal storage pool capacity. The capacity similarity difference characterizes the similarity difference between the storage pool capacities of different normal storage pools when they are within the same data capacity. The list determination module filters the normal storage pools in the initial storage pool list that are within the capacity similarity difference based on the benchmark storage pool to obtain a target storage list. The file storage module determines the target storage pool for storing the file to be written from the target storage list based on the file information of the file to be written. Through the above process, the amount of disk data in the storage pool can be reduced, and the system can be managed separately to ensure stability. At the same time, after the storage pool is grouped according to the nodes, a new fault domain is formed. The fault recovery and handling of a single node failure is carried out within the storage pool without affecting other storage pools, effectively isolating the faulty node and reducing the impact of storage pool failure on business.

[0029] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the data processing method described in the first aspect or any corresponding embodiment thereof.

[0030] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the data processing method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0031] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of an application environment according to an embodiment of the present invention;

[0033] Figure 2 This is a flowchart illustrating the data processing method according to an embodiment of the present invention;

[0034] Figure 3 This is a flowchart illustrating another data processing method according to an embodiment of the present invention;

[0035] Figure 4 This is a flowchart illustrating another data processing method according to an embodiment of the present invention;

[0036] Figure 5 This is a flowchart illustrating another data processing method according to an embodiment of the present invention;

[0037] Figure 6 This is a data processing logic diagram of the unstructured distributed storage according to an embodiment of the present invention;

[0038] Figure 7 This is a structural block diagram of the data processing device according to an embodiment of the present invention;

[0039] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] The terms "first" and "second" in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses. The term "multiple" in this invention can mean at least two, for example, two, three, or more, and the embodiments of this invention are not limited thereto.

[0042] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided by an embodiment of the present invention. The diagram includes a storage server 100, which may contain a processor 101 and a memory 102. The storage server 100 can communicate with a storage management server 200 via a network 300. The storage management server 200 can provide services (such as management services) to computing programs installed on clients. A database 201 can be set up on or independently of the storage management server 200 to provide data storage services to the storage management server 200. Furthermore, the storage management server 200 may run a processing engine 202, which can be used to execute steps performed by the storage management server 200.

[0043] Optionally, the storage server 100 may be, but is not limited to, a terminal capable of computing data, such as a mobile terminal (e.g., a tablet), a laptop, or a PC (Personal Computer). The network may include, but is not limited to, a wireless network or a wired network. The wireless network includes Bluetooth, Wi-Fi (Wireless Fidelity), and other networks that enable wireless communication. The wired network may include, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a storage management server cluster. The storage management server 200 may be, but is not limited to, any hardware device capable of computing.

[0044] Furthermore, in this embodiment, the above-described data processing method can also be applied to, but is not limited to, a powerful independent processing device without requiring data interaction. For example, the processing device can be, but is not limited to, a powerful terminal device; that is, the various operations in the above-described data processing method can be integrated into a single independent processing device. The above is merely an example, and no limitation is made in this embodiment.

[0045] Optionally, in this embodiment, the above-described data processing method can be executed by the storage management server 200, by the storage server 100, or jointly by the storage management server 200 and the storage server 100. Alternatively, the data processing method of this embodiment can be executed by a client installed on the storage server 100.

[0046] According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] This embodiment provides a data processing method that can be used in the aforementioned storage server. The storage server includes a namespace, which is bound to a storage pool to facilitate determining the storage pool to which the file to be written is written based on the namespace. Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0048] Step S201: Obtain the file to be written and the initial storage pool list. The initial storage pool list contains multiple storage nodes, and each storage node stores at least one normal storage pool. The normal storage pools are isolated from each other according to the storage nodes.

[0049] As described above, by obtaining the file to be written and the initial storage pool list, a target storage pool for storing the written file can be selected from the normal storage pools within the multiple storage nodes of the initial storage pool list. This ensures resource balance among the storage pools and guarantees stable system operation. Furthermore, since the normal storage pools are isolated according to storage nodes, single-node failures are handled within the storage pool itself, without affecting other storage pools. Newly created files from the metadata service are written to the normal storage pool, effectively isolating failed nodes and reducing the impact on business operations.

[0050] In some optional implementations, a unique file identifier can be created based on a file write request, and then a file to be written can be created based on this unique file identifier to perform file data writing. When obtaining the initial storage pool list, the storage node to which each data disk belongs can be obtained; storage pools can be allocated to the data disks based on the storage nodes; the original storage pool list can be determined based on the storage pools corresponding to the storage nodes; and the initial storage pool list can be determined based on the storage status of each storage pool in the original storage pool list, that is, the list consisting of storage pools with normal storage status can be determined as the initial storage pool list.

[0051] Step S202: For any normal storage pool, identify the storage pool capacity of the normal storage pool, and determine the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the same data capacity.

[0052] As shown above, by identifying the storage pool capacity of the normal storage pool and determining the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the benchmark storage pool and the capacity similarity difference, thus avoiding the instability of the file system due to the imbalance of data storage between various storage pools.

[0053] In some alternative implementations, the normal storage pool with the smallest storage pool capacity in the original storage pool list can be obtained first; the normal storage pool with the smallest storage pool capacity can be determined as the baseline storage pool; and the capacity similarity difference can be determined based on the storage pool capacity difference among the various normal storage pools.

[0054] Specifically, the storage capacity of each normal storage pool in the initial storage pool list can be monitored to obtain the current storage capacity of each normal storage pool. The current storage capacities of each normal storage pool can then be compared to determine the normal storage pool with the smallest capacity, which is then designated as the baseline storage pool. Alternatively, the storage capacities of each normal storage pool can be ranked according to their largest capacity, and the capacities of the first and last normal storage pools can be subtracted sequentially to obtain a capacity difference dataset. The average of the capacity differences in this dataset is then multiplied by an adjustment factor to obtain the capacity similarity difference. This adjustment factor can be obtained by multiplying the error coefficient by the total failure coefficient of the storage pool. Optionally, this capacity similarity difference can also be obtained by adjusting the capacity differences in the capacity difference dataset based on a user-defined storage pool balance and then averaging the results.

[0055] In some optional implementations, the normal storage pool with the largest remaining capacity in the original storage pool list can be obtained first, and then directly designated as the target storage pool; alternatively, the normal storage pool with the smallest remaining capacity in the original storage pool list can be obtained, and then directly designated as the target storage pool. Optionally, the normal storage pool with the largest remaining capacity can also be designated as the baseline pool.

[0056] Step S203: Based on the baseline storage pool, filter the normal storage pools in the initial storage pool list that are within the same capacity range to obtain the target storage list.

[0057] As shown above, by filtering normal storage pools within a similar capacity range in the initial storage pool list based on the baseline storage pool, multiple normal storage pools with similar capacities can be selected from the initial storage pool list. The target storage list is then obtained based on these multiple normal storage pools with similar capacities, which improves the efficiency of determining the target storage pool and ensures the balance of resources among the normal storage pools.

[0058] In some optional implementations, the storage pool capacity of each normal storage pool in the initial storage pool list can be obtained first; the capacity difference between the storage pool capacity of the normal storage pool and the storage pool capacity of the baseline storage pool can be calculated; the capacity difference can be compared with the capacity similarity difference, and the normal storage pools in the initial storage pool list can be filtered based on the comparison results to obtain the target storage list.

[0059] When filtering normal storage pools in the initial storage pool list based on the comparison results to obtain the target storage list, normal storage pools can be filtered out from the initial storage pool list if the capacity difference represented by the comparison results is less than or equal to the capacity similarity difference; the target storage list is obtained based on the filtered normal storage pools.

[0060] Specifically, when obtaining the target storage list, the storage pool capacity of each normal storage pool in the initial storage pool list can be obtained first. The storage pool capacity of the normal storage pool is then subtracted from the storage pool capacity of the baseline storage pool to obtain the capacity difference. This capacity difference is then compared with the capacity difference of similar pools, and normal storage pools with a capacity difference less than or equal to the capacity difference of similar pools are filtered out from the initial storage pool list. The target storage list is obtained based on the filtered normal storage pools.

[0061] Step S204: Based on the file information of the file to be written, determine the target storage pool for storing the file to be written from the target storage list.

[0062] In this embodiment, the target storage pool for storing the file to be written is determined from the target storage list based on the file information of the file to be written. This allows the selected target storage pool to be recorded in the metadata attributes of the file to be written, facilitating subsequent data writing to the specified target storage pool. Optionally, the normal storage pool with the smallest capacity can also be directly used as the target storage pool.

[0063] The data processing method provided in this embodiment first obtains the file to be written and an initial storage pool list. This allows for the selection of a target storage pool from the normal storage pools in the initial storage pool list to store the file to be written, thereby ensuring resource balance among the storage pools and guaranteeing stable system operation. Next, by identifying the storage pool capacity of the normal storage pools and determining a baseline storage pool and a capacity similarity difference based on these capacities, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the baseline storage pool and the capacity similarity difference. This avoids file system instability caused by data storage imbalance among the storage pools. This invention utilizes a baseline storage pool to filter normal storage pools within a similar capacity range from the initial storage pool list. This allows for the selection of multiple normal storage pools with similar capacities from the initial list, and the generation of a target storage pool list based on these pools. This improves the efficiency of target storage pool determination and ensures resource balance among normal storage pools. Furthermore, by using the file information of the file to be written, the target storage pool for storing the file is determined from the target storage list. The selected target storage pool is then recorded in the metadata attributes of the file to be written, facilitating subsequent data writing to the specified target storage pool. Therefore, this invention reduces disk data within storage pools, manages it systematically, and ensures system stability. Simultaneously, by grouping storage pools by nodes to form new fault domains, single-node failures are handled and recovered within the storage pool itself, without affecting other storage pools. This effectively isolates faulty nodes and reduces the impact of storage pool failures on business operations.

[0064] This embodiment provides a data processing method that can be used in the aforementioned storage server. The storage server includes a namespace, which is bound to a storage pool to facilitate determining the storage pool to which the file to be written is written based on the namespace. Figure 3 This is a flowchart of a data processing method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0065] Step S301: Obtain the file to be written and the initial storage pool list. The initial storage pool list contains multiple storage nodes, and each storage node stores at least one normal storage pool. The normal storage pools are isolated from each other according to the storage nodes.

[0066] As described above, by obtaining the file to be written and the initial storage pool list, a target storage pool for storing the written file can be selected from the normal storage pools within the multiple storage nodes of the initial storage pool list. This ensures resource balance among the storage pools and guarantees stable system operation. Furthermore, since the normal storage pools are isolated according to storage nodes, single-node failures are handled within the storage pool itself, without affecting other storage pools. Newly created files from the metadata service are written to the normal storage pool, effectively isolating failed nodes and reducing the impact on business operations.

[0067] Specifically, step S301 above includes:

[0068] Step S3011: Obtain the storage node to which each data disk belongs.

[0069] As shown above, by obtaining the storage node to which each data disk belongs, it is possible to divide the storage pool based on the storage node, and to achieve data disk isolation between different storage nodes, thereby reducing the failure domain of the storage node.

[0070] In some alternative implementations, the storage nodes to which data disks belong can be assigned based on the number of data disks, the capacity of the data disks, or the number, performance, and capacity of the data disks.

[0071] When allocating storage nodes based on the number of data disks, multiple data disks can be evenly distributed across storage nodes based on the number of storage nodes. When allocating storage nodes based on the capacity of data disks, multiple data disks can be allocated across storage nodes based on both the number of storage nodes and the capacity of each data disk, ensuring that the total storage capacity across storage nodes is similar. When allocating storage nodes based on the number of data disks, their performance, and their capacity, a first-number group of data disks with the first-sized capacity can be allocated to one storage node, and a second-number group of data disks with the second-sized capacity can be allocated to another storage node, thus ensuring a balance in storage capacity across storage nodes.

[0072] Understandably, when the number and capacity of data disks do not meet the requirements of a relatively balanced allocation, one storage node can be given a larger storage capacity, and this storage node can be used for data writing in the future. Alternatively, a data disk that does not meet the requirements of a balanced allocation can be designated as an independent storage node to store a smaller amount of data.

[0073] Step S3012: Divide the data disk into corresponding storage pools based on the storage nodes.

[0074] As shown above, by dividing the data disk into corresponding storage pools based on storage nodes, data isolation and fault isolation between different storage pools can be achieved, so that the failure of a single storage node can be recovered and processed within the storage pool without affecting other storage pools.

[0075] In some alternative implementations, multiple storage nodes can be evenly distributed across multiple storage pools to reduce the size of a single pool, as a large single pool results in significant internal connection overhead. Therefore, a grouped multi-pool technique is used, for example, creating a storage pool for every 10 storage nodes. A cluster of 30 nodes can be divided into 3 storage pools, and these 3 storage pools are bound to a namespace to facilitate the determination of the target storage pool for storing files to be written.

[0076] Step S3013: Determine the original list of storage pools based on the storage pools corresponding to the storage nodes.

[0077] As shown above, the original storage pool list is determined based on the storage pool corresponding to the storage node, so as to facilitate the determination of the initial storage pool list based on the original storage pool list.

[0078] Step S3014: Determine the initial storage pool list based on the storage status of each storage pool in the original storage pool list.

[0079] As shown above, an initial storage pool list is determined based on the storage status of each storage pool in the original storage pool list, so as to determine the target storage pool for storing the file to be written based on the initial storage pool list.

[0080] In some optional implementations, the service status of each storage pool in the initial storage pool list can be monitored to obtain the storage pool status of each storage pool. When the storage pool status indicates that the service status of the storage pool is abnormal, the storage pools with abnormal service status in the original storage pool list are removed to obtain the initial storage pool list. An abnormal service status can be caused by the number of faulty storage nodes in the storage pool reaching a fault threshold, or by the storage pool becoming unserviceable. Unserviceable indicates that the current storage node or storage pool cannot complete the data writing within the threshold time.

[0081] In some optional implementations, when determining the storage pool status of each storage pool, target data can be used to perform read / write tests on the data disks within each storage node of the storage pool to obtain test results. When the test results indicate that the target data read / write is successful, a "test successful" tag is added to each storage node to indicate that the current storage node's service status is normal; when the test results indicate that the target data read / write fails, a "test failed" tag is added to each storage node to indicate that the current storage node's service status is abnormal. The read / write test on the target data can be a continuous data read / write test or a random data read / write test.

[0082] Step S302: For any normal storage pool, identify the storage pool capacity of the normal storage pool, and determine the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the same data capacity.

[0083] As shown above, by identifying the storage pool capacity of the normal storage pool and determining the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the benchmark storage pool and the capacity similarity difference, thus avoiding the instability of the file system due to the imbalance of data storage between storage pools.

[0084] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0085] Step S303: Based on the baseline storage pool, filter the normal storage pools in the initial storage pool list that are within the same capacity range to obtain the target storage list.

[0086] As shown above, by filtering normal storage pools within a similar capacity range in the initial storage pool list based on the baseline storage pool, multiple normal storage pools with similar capacities can be selected from the initial storage pool list. The target storage list is then obtained based on these multiple normal storage pools with similar capacities, which improves the efficiency of determining the target storage pool and ensures the balance of resources among the normal storage pools.

[0087] Please see details Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0088] Step S304: Based on the file information of the file to be written, determine the target storage pool for storing the file to be written from the target storage list.

[0089] In this embodiment, the target storage pool for storing the file to be written is determined from the target storage list based on the file information of the file to be written, and then the selected target storage pool is recorded in the metadata attribute of the file to be written, so that subsequent data can be written to the target storage pool specified by the file to be written.

[0090] Please see details Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0091] The data processing method provided in this embodiment first obtains the file to be written and an initial storage pool list, so as to select a target storage pool for storing the file to be written from the normal storage pools in the initial storage pool list, thereby ensuring resource balance among the storage pools and ensuring stable system operation; by identifying the storage pool capacity of the normal storage pool, and determining the benchmark storage pool and capacity similarity difference based on the storage pool capacity of the normal storage pool, the target storage pool for storing the file to be written is selected from the initial storage pool list based on the benchmark storage pool and capacity similarity difference, avoiding the instability of the file system operation due to data storage imbalance among the storage pools; By filtering normal storage pools within a similar capacity range from the initial storage pool list based on a baseline storage pool, multiple normal storage pools with similar capacities can be selected from the initial list. A target storage list is then derived based on these pools, improving the efficiency of target storage pool determination and ensuring resource balance among normal storage pools. Furthermore, by using the file information of the file to be written, the target storage pool for storing the file is determined from the target storage list. This selected target storage pool is then recorded in the metadata attributes of the file to be written, facilitating subsequent data writing to the designated target storage pool. Therefore, this invention reduces disk data within storage pools, manages it systematically, and ensures system stability. Simultaneously, by grouping storage pools by nodes to form new fault domains, single-node failures are handled and recovered within the storage pool itself, without affecting other storage pools, effectively isolating faulty nodes and reducing the impact of storage pool failures on business operations.

[0092] This embodiment provides a data processing method that can be used in the aforementioned storage server. The storage server includes a namespace, which is bound to a storage pool to facilitate determining the storage pool to which the file to be written is written based on the namespace. Figure 4 This is a flowchart of a data processing method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0093] Step S401: Obtain the file to be written and the initial storage pool list. The initial storage pool list contains multiple storage nodes, and each storage node stores at least one normal storage pool. The normal storage pools are isolated from each other according to the storage nodes.

[0094] As described above, by obtaining the file to be written and the initial storage pool list, a target storage pool for storing the written file can be selected from the normal storage pools within the multiple storage nodes of the initial storage pool list. This ensures resource balance among the storage pools and guarantees stable system operation. Furthermore, since the normal storage pools are isolated according to storage nodes, single-node failures are handled within the storage pool itself, without affecting other storage pools. Newly created files from the metadata service are written to the normal storage pool, effectively isolating failed nodes and reducing the impact on business operations.

[0095] Please see details Figure 3 Step S301 of the illustrated embodiment will not be described again here.

[0096] Step S402: For any normal storage pool, identify the storage pool capacity of the normal storage pool, and determine the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the data capacity.

[0097] As shown above, by identifying the storage pool capacity of the normal storage pool and determining the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the benchmark storage pool and the capacity similarity difference, thus avoiding the instability of the file system due to the imbalance of data storage between storage pools.

[0098] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0099] Step S403: Based on the baseline storage pool, filter the normal storage pools in the initial storage pool list that are within the same capacity range to obtain the target storage list.

[0100] As shown above, by filtering normal storage pools within a similar capacity range in the initial storage pool list based on the baseline storage pool, multiple normal storage pools with similar capacities can be selected from the initial storage pool list. The target storage list is then obtained based on these multiple normal storage pools with similar capacities, which improves the efficiency of determining the target storage pool and ensures the balance of resources among the normal storage pools.

[0101] Specifically, step S403 includes:

[0102] Step S4031: Obtain the storage pool capacity of each normal storage pool in the initial storage pool list.

[0103] As shown above, by obtaining the storage pool capacity of each normal storage pool in the initial storage pool list, the target storage list can be determined based on the storage capacity of each normal storage pool.

[0104] Step S4032: Calculate the capacity difference between the storage pool capacity of the normal storage pool and the storage pool capacity of the baseline storage pool.

[0105] As shown above, by calculating the capacity difference between the storage pool capacity of the normal storage pool and the storage pool capacity of the benchmark storage pool, a normal storage pool with similar storage capacity can be determined based on this capacity difference.

[0106] In some alternative implementations, the storage pool capacity of each normal storage pool in the initial storage pool list can be obtained first, and the storage pool capacity of the normal storage pool can be subtracted from the storage pool capacity of the baseline storage pool to obtain the capacity difference.

[0107] Step S4033: Compare the capacity difference with the capacity similarity difference, and filter the normal storage pools in the initial storage pool list based on the comparison results to obtain the target storage list.

[0108] As shown above, by comparing the capacity difference with the capacity similarity difference, and filtering the normal storage pools in the initial storage pool list based on the comparison results, the target storage list can be obtained based on multiple normal storage pools with similar capacity, thereby improving the efficiency of determining the target storage pool and ensuring the balance of resources among normal storage pools.

[0109] In some optional implementations, when the comparison results indicate that the capacity difference is less than or equal to the capacity similarity difference, the normal storage pool is filtered out from the initial storage pool list; the target storage list is obtained based on the filtered normal storage pools.

[0110] Step S404: Based on the file information of the file to be written, determine the target storage pool for storing the file to be written from the target storage list.

[0111] In this embodiment, the target storage pool for storing the file to be written is determined from the target storage list based on the file information of the file to be written, and then the selected target storage pool is recorded in the metadata attribute of the file to be written, so that subsequent data can be written to the target storage pool specified by the file to be written.

[0112] Please see details Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0113] The data processing method provided in this embodiment first obtains the file to be written and an initial storage pool list. This allows for the selection of a target storage pool from the normal storage pools in the initial storage pool list to store the file to be written, thereby ensuring resource balance among the storage pools and guaranteeing stable system operation. Next, by identifying the storage pool capacity of the normal storage pools and determining a baseline storage pool and a capacity similarity difference based on these capacities, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the baseline storage pool and the capacity similarity difference. This avoids file system instability caused by data storage imbalance among the storage pools. This invention utilizes a baseline storage pool to filter normal storage pools within a similar capacity range from the initial storage pool list. This allows for the selection of multiple normal storage pools with similar capacities from the initial list, and the generation of a target storage pool list based on these pools. This improves the efficiency of target storage pool determination and ensures resource balance among normal storage pools. Furthermore, by using the file information of the file to be written, the target storage pool for storing the file is determined from the target storage list. The selected target storage pool is then recorded in the metadata attributes of the file to be written, facilitating subsequent data writing to the specified target storage pool. Therefore, this invention reduces disk data within storage pools, manages it systematically, and ensures system stability. Simultaneously, by grouping storage pools by nodes to form new fault domains, single-node failures are handled and recovered within the storage pool itself, without affecting other storage pools. This effectively isolates faulty nodes and reduces the impact of storage pool failures on business operations.

[0114] This embodiment provides a data processing method that can be used in the aforementioned storage server. The storage server includes a namespace, which is bound to a storage pool to facilitate determining the storage pool to which the file to be written is written based on the namespace. Figure 5 This is a flowchart of a data processing method according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0115] Step S501: Obtain the file to be written and the initial storage pool list. The initial storage pool list contains multiple storage nodes, and each storage node stores at least one normal storage pool. The normal storage pools are isolated from each other according to the storage nodes.

[0116] As described above, by obtaining the file to be written and the initial storage pool list, a target storage pool for storing the written file can be selected from the normal storage pools within the multiple storage nodes of the initial storage pool list. This ensures resource balance among the storage pools and guarantees stable system operation. Furthermore, since the normal storage pools are isolated according to storage nodes, single-node failures are handled within the storage pool itself, without affecting other storage pools. Newly created files from the metadata service are written to the normal storage pool, effectively isolating failed nodes and reducing the impact on business operations.

[0117] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0118] Step S502: For any normal storage pool, identify the storage pool capacity of the normal storage pool, and determine the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the data capacity.

[0119] As shown above, by identifying the storage pool capacity of the normal storage pool and determining the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the benchmark storage pool and the capacity similarity difference, thus avoiding the instability of the file system due to the imbalance of data storage between storage pools.

[0120] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0121] Step S503: Based on the baseline storage pool, filter the normal storage pools in the initial storage pool list that are within the same capacity range to obtain the target storage list.

[0122] As shown above, by filtering normal storage pools within a similar capacity range in the initial storage pool list based on the baseline storage pool, multiple normal storage pools with similar capacities can be selected from the initial storage pool list. The target storage list is then obtained based on these multiple normal storage pools with similar capacities, which improves the efficiency of determining the target storage pool and ensures the balance of resources among the normal storage pools.

[0123] Please see details Figure 4 Step S403 of the illustrated embodiment will not be described again here.

[0124] Step S504: Based on the file information of the file to be written, determine the target storage pool for storing the file to be written from the target storage list.

[0125] In this embodiment, the target storage pool for storing the file to be written is determined from the target storage list based on the file information of the file to be written, and then the selected target storage pool is recorded in the metadata attribute of the file to be written, so that subsequent data can be written to the target storage pool specified by the file to be written.

[0126] Specifically, step S504 above includes:

[0127] Step S5041: Obtain the file information of the file to be written, including the file name or file path.

[0128] As shown above, by obtaining the file information of the file to be written, the target storage pool can be determined based on the file name or file path of the file to be written.

[0129] Step S5042: Perform a hash operation on the file information to obtain a hash value.

[0130] As shown above, by performing a hash operation on the filename or file path of the file to be written, a unique target storage pool can be determined from the target storage list, ensuring the uniqueness of the storage pool acquisition.

[0131] Step S5043: The hash value is moduloed based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written.

[0132] As shown above, by hashing the filename of the file to be written to a unique storage pool in the target storage pool list, the reliability of allocating a storage pool for the file to be written is ensured.

[0133] In some optional implementations, the filename of the file to be written can be obtained first; a hash operation can be performed on the filename to obtain a hash value; and the hash value can be moduloed based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written. It is understood that by hashing the file path of the file to be written to a unique storage pool in the target storage pool list, the reliability of allocating a storage pool for the file to be written is ensured.

[0134] In some optional implementations, the file path of the file to be written can be obtained first; a hash operation can be performed on the file path to obtain a hash value; the hash value can then be moduloed based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written. It is understood that by hashing the file path of the file to be written to a unique storage pool in the target storage pool list, the reliability of allocating a storage pool for the file to be written is ensured. Alternatively, a storage pool can be selected from the target storage list using a random function to determine it as the target storage pool.

[0135] In some optional implementations, the content corresponding to the file information of the file to be written can be encoded to obtain the content feature vector of the file to be written, and the content feature vector can be moduloed and divided by the maximum number of the target storage pool list to obtain the target storage pool for storing the file to be written.

[0136] The data processing method provided in this embodiment first obtains the file to be written and an initial storage pool list. This allows for the selection of a target storage pool from the normal storage pools in the initial storage pool list to store the file to be written, thereby ensuring resource balance among the storage pools and guaranteeing stable system operation. Next, by identifying the storage pool capacity of the normal storage pools and determining a baseline storage pool and a capacity similarity difference based on these capacities, the target storage pool for storing the file to be written can be selected from the initial storage pool list based on the baseline storage pool and the capacity similarity difference. This avoids file system instability caused by data storage imbalance among the storage pools. This invention utilizes a baseline storage pool to filter normal storage pools within a similar capacity range from the initial storage pool list. This allows for the selection of multiple normal storage pools with similar capacities from the initial list, and the generation of a target storage pool list based on these pools. This improves the efficiency of target storage pool determination and ensures resource balance among normal storage pools. Furthermore, by using the file information of the file to be written, the target storage pool for storing the file is determined from the target storage list. The selected target storage pool is then recorded in the metadata attributes of the file to be written, facilitating subsequent data writing to the specified target storage pool. Therefore, this invention reduces disk data within storage pools, manages it systematically, and ensures system stability. Simultaneously, by grouping storage pools by nodes to form new fault domains, single-node failures are handled and recovered within the storage pool itself, without affecting other storage pools. This effectively isolates faulty nodes and reduces the impact of storage pool failures on business operations.

[0137] In some alternative implementations, the logic for deploying unstructured distributed storage namespace multi-pool storage is as follows: Figure 6 As shown,

[0138] First, data pools are created for different node groups, and nodes are isolated between data pools. Namespaces are bound to storage pools, and information such as specified storage pools and policies are persisted to the metadata storage pool by configuring management nodes. Monitoring nodes collect the status of each storage pool periodically and synchronize it to the metadata service. When the metadata service starts, it loads the configuration information and determines the final storage pool of the file based on the monitoring status and configuration policy when processing creation requests, through file ownership calculation.

[0139] Furthermore, for monitoring nodes: periodically send query requests to the storage pool, and the storage pool summarizes its own status and returns it to the monitoring node. The summarized information mainly includes the storage pool level and storage pool status. After collecting the storage pool information, the monitoring node synchronizes the basic information of the storage pool to the metadata service.

[0140] Furthermore, for the configuration management node: users configure namespace multi-storage pool policies; namespace multi-storage pool configuration mainly includes a list of storage pools specified by the namespace, configuration policies, and capacity differences, in GB. The configuration policies mainly include capacity policies and assigning data to storage pools with lower capacity to ensure capacity balance across storage pools; other policies can be supported later. The configuration policies are serialized and then stored in the metadata storage pool; finally, the configuration is refreshed to the metadata server.

[0141] Furthermore, regarding the metadata service, upon receiving a business request to write a file, after creating a unique file identifier based on the request, the file ownership calculation begins. First, the metadata service, based on storage pool status information pushed by monitoring nodes, excludes abnormal storage pools; selects the storage pool with the smallest capacity; and filters a list of storage pools with similar capacities from the normal storage pool list based on the capacity difference. Then, based on the filename, it hashes the specified unique storage pool (the target storage pool) from the list of storage pools with similar capacities, binds the target storage pool to file attributes, and persists it to the metadata pool. Subsequent file data is then written to the specified data pool.

[0142] This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0143] This embodiment provides a data processing device, such as... Figure 7 As shown, it includes:

[0144] The information acquisition module 701 is used to acquire the file to be written and the initial storage pool list. The initial storage pool list contains multiple storage nodes, and each storage node stores at least one normal storage pool. The normal storage pools are isolated from each other according to the storage nodes.

[0145] The information determination module 702 is used to identify the storage pool capacity of any normal storage pool, and determine the benchmark storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the same data capacity.

[0146] The list determination module 703 is used to filter normal storage pools within a similar capacity range in the initial storage pool list based on the baseline storage pool to obtain the target storage list.

[0147] File storage module 704 is used to determine the target storage pool for storing the file to be written from the target storage list based on the file information of the file to be written.

[0148] In some alternative implementations, the file storage module 704 includes:

[0149] The file information acquisition unit is used to acquire the file information of the file to be written, including the file name or file path.

[0150] The hash operation unit is used to perform hash operations on file information to obtain hash values.

[0151] The storage pool determination unit is used to perform modulo operation on the hash value based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written.

[0152] In some alternative implementations, the list determination module 703 includes:

[0153] The storage pool capacity acquisition unit is used to acquire the storage pool capacity of each normal storage pool in the initial storage pool list.

[0154] The capacity difference calculation unit is used to calculate the capacity difference between the storage pool capacity of the normal storage pool and the storage pool capacity of the reference storage pool.

[0155] The difference comparison unit is used to compare the capacity difference with the capacity similarity difference, and based on the comparison results, to filter the normal storage pools in the initial storage pool list to obtain the target storage list.

[0156] In some optional implementations, the difference comparison unit includes:

[0157] The storage pool filtering subunit is used to filter out normal storage pools from the initial storage pool list when the capacity difference in the comparison results is less than or equal to the capacity similarity difference.

[0158] The storage list determination sub-unit is used to obtain the target storage list based on the selected normal storage pool.

[0159] In some alternative implementations, the information determination module 702 includes:

[0160] The storage pool acquisition unit is used to acquire the normal storage pool with the smallest storage pool capacity from the initial storage pool list.

[0161] The baseline storage pool determination unit is used to determine the normal storage pool with the smallest storage pool capacity as the baseline storage pool.

[0162] The similarity difference determination unit is used to determine the similarity difference in capacity based on the capacity difference of each normal storage pool.

[0163] In some optional implementations, the information acquisition module 701 includes:

[0164] The storage node acquisition unit is used to acquire the storage node to which each data disk belongs.

[0165] The storage pool partitioning unit is used to partition the data disk into corresponding storage pools based on the storage node.

[0166] The first list determination unit is used to determine the original list of storage pools based on the storage pools corresponding to the storage nodes.

[0167] The second list determination unit is used to determine the initial storage pool list based on the storage status of each storage pool in the original storage pool list.

[0168] In some optional implementations, the second list determination unit includes:

[0169] The storage pool status monitoring subunit is used to monitor the service status of the storage pool and obtain the storage pool status.

[0170] The storage pool list determination sub-unit is used to remove storage pools with abnormal service status from the original storage pool list when the storage pool status characterization of the storage pool service status is abnormal, so as to obtain the initial storage pool list.

[0171] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0172] In this embodiment, the data processing device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0173] This invention also provides a computer device having the above-described features. Figure 7 The data processing device shown.

[0174] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., identified as a storage server array, a group of blade storage servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0175] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0176] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0177] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, server clusters, mobile communication networks, and combinations thereof.

[0178] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0179] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0180] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0181] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to unstructured storage systems, the method includes: Obtain the file to be written and the initial storage pool list. The initial storage pool list stores multiple storage nodes. Each storage node stores at least one normal storage pool, and the normal storage pools are isolated from each other according to the storage nodes. For any of the normal storage pools, the storage pool capacity of the normal storage pool is identified, and a benchmark storage pool and a capacity similarity difference are determined based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the same data capacity. The capacity similarity difference is obtained by multiplying the mean of the storage pool capacity difference by an adjustment factor. The adjustment factor is obtained by multiplying the error factor by the total failure factor of the storage pool. Based on the baseline storage pool, normal storage pools in the initial storage pool list that are within the same capacity difference are filtered to obtain the target storage list; Based on the file information of the file to be written, a target storage pool for storing the file to be written is determined from the target storage list, including: Obtain the file information of the file to be written, the file information including the file name or file path; Perform a hash operation on the file information to obtain a hash value; The hash value is moduloed based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written, and the target storage pool is recorded in the metadata attribute of the file to be written.

2. The method according to claim 1, characterized in that, The step involves filtering normal storage pools within the similar capacity range in the initial storage pool list based on the baseline storage pool to obtain a target storage list, including: Obtain the storage pool capacity of each normal storage pool in the initial storage pool list; Calculate the capacity difference between the storage pool capacity of the normal storage pool and the storage pool capacity of the reference storage pool; The capacity difference is compared with the capacity similarity difference, and the normal storage pools in the initial storage pool list are filtered based on the comparison results to obtain the target storage list.

3. The method according to claim 2, characterized in that, The process of filtering the normal storage pools in the initial storage pool list based on the comparison results to obtain the target storage list includes: When the comparison result indicates that the capacity difference is less than or equal to the capacity similarity difference, the normal storage pool is selected from the initial storage pool list; The target storage list is obtained based on the selected normal storage pools.

4. The method according to claim 1, characterized in that, The determination of the baseline storage pool and the capacity similarity difference based on the storage pool capacity of the normal storage pool includes: Obtain the normal storage pool with the smallest storage pool capacity from the initial storage pool list; The normal storage pool with the smallest storage pool capacity is determined as the baseline storage pool; The capacity similarity difference is determined based on the capacity difference of each normal storage pool.

5. The method according to claim 1, characterized in that, Retrieve the initial list of storage pools, including: Obtain the storage node to which each data disk belongs; Based on the storage node, a corresponding storage pool is allocated for the data disk; Based on the storage pool corresponding to the storage node, determine the original list of storage pools; The initial storage pool list is determined based on the storage status of each storage pool in the original storage pool list.

6. The method according to claim 5, characterized in that, The initial storage pool list is determined based on the storage status of each storage pool in the original storage pool list; The service status of the storage pool is monitored to obtain the storage pool status; When the storage pool status indicates that the service status of the storage pool is abnormal, the storage pools with abnormal service status in the original storage pool list are removed to obtain the initial storage pool list.

7. A data processing apparatus, characterized in that, The device, used in unstructured storage systems, includes: The information acquisition module is used to acquire the file to be written and the initial storage pool list. The initial storage pool list stores multiple storage nodes, and each storage node stores at least one normal storage pool. The normal storage pools are isolated from each other according to the storage nodes. The information determination module is used to identify the storage pool capacity of any normal storage pool, and determine a benchmark storage pool and a capacity similarity difference based on the storage pool capacity of the normal storage pool. The capacity similarity difference is used to characterize the similarity difference between the storage pool capacities of each normal storage pool and the same data capacity. The capacity similarity difference is obtained by multiplying the mean of the storage pool capacity difference by an adjustment coefficient. The adjustment coefficient is obtained by multiplying the error coefficient by the total failure coefficient of the storage pool. The list determination module is used to filter normal storage pools in the initial storage pool list that are within the same capacity difference based on the baseline storage pool to obtain the target storage list; The file storage module is used to determine a target storage pool for storing the file to be written from the target storage list based on the file information of the file to be written, including: Obtain the file information of the file to be written, including the file name or file path; perform a hash operation on the file information to obtain a hash value; take the modulo of the hash value based on the number of each storage pool in the target storage list to obtain the target storage pool for storing the file to be written, and record the target storage pool in the metadata attribute of the file to be written.

8. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 6.

Images