Storage defragmentation processing method, device and equipment based on cloud storage system and medium

Abstract

The application discloses a storage fragment processing method and device based on a cloud storage system, equipment and a medium, relates to the technical field of cloud computing and data processing, and can be applied to the fields of finance and medicine and comprises the following steps: acquiring multidimensional feature data of each storage fragment in a target storage space; determining the fragment types to which each storage fragment belongs based on the multidimensional feature data of each storage fragment; generating a target arrangement strategy for each fragment type according to the fragment types to which each storage fragment belongs; and performing an arrangement operation on each storage fragment in the target storage space according to the generated target arrangement strategy. The scheme effectively solves the problem that the arrangement effect is poor due to the use of a single threshold arrangement strategy in the prior art, and improves the arrangement effect of storage fragments.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing technology and data processing technology, and can be applied to the financial and medical fields. In particular, it relates to a storage fragmentation processing method, apparatus, equipment and medium based on a cloud storage system. Background Technology

[0002] With the rapid development of cloud computing technology, cloud storage systems (such as object storage services like Amazon S3) have become the core infrastructure for storing massive amounts of unstructured data, and are widely used in scenarios such as financial transaction record storage and medical image archiving. During long-term operation, these systems generate a large amount of storage fragmentation due to the frequent creation, modification, and deletion of data, which in turn affects storage space utilization efficiency and data access performance.

[0003] To address this issue, existing technologies typically employ automated defragmentation schemes based on predefined rules. For example, they rely on a single object size threshold as a criterion; when multiple storage fragments smaller than the preset threshold are detected, a unified defragmentation process is triggered. However, in real-world business scenarios such as finance and healthcare, different storage fragments exhibit significant differences in access frequency, lifecycle status, and other characteristics. Applying the same defragmentation strategy to frequently accessed transaction log fragments and infrequently archived medical image fragments can easily lead to performance fluctuations or resource waste. This "one-size-fits-all" static processing mode results in a lack of targeted defragmentation, leading to poor defragmentation results. Therefore, improving the defragmentation effect of storage fragments has become an urgent problem to be solved in this field. Summary of the Invention

[0004] This invention provides a storage fragmentation processing method, apparatus, device, and medium based on a cloud storage system to solve the problem of poor defragmentation effect caused by the use of a single threshold defragmentation strategy in the prior art.

[0005] Firstly, a storage fragmentation processing method based on a cloud storage system is provided, including: Obtain multi-dimensional feature data of each storage fragment in the target storage space; Based on the multi-dimensional feature data of each storage fragment, the fragment type of each storage fragment is determined; Based on the fragment type to which each of the storage fragments belongs, generate a target compaction strategy for each of the fragment types; Based on the generated target defragmentation strategy, defragmentation operations are performed on each storage fragment of the target storage space.

[0006] Secondly, a storage fragmentation processing device based on a cloud storage system is provided, comprising: The first acquisition module is used to acquire multi-dimensional feature data of each storage fragment in the target storage space; The determination module is used to determine the fragment type of each storage fragment based on the multi-dimensional feature data of each storage fragment; The generation module is used to generate a target compaction strategy for each of the storage fragments based on the fragment type to which each of the storage fragments belongs; The execution module is used to perform reorganization operations on each storage fragment of the target storage space according to the generated target reorganization strategy.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described storage fragmentation processing method based on a cloud storage system.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described storage fragmentation processing method based on a cloud storage system.

[0009] The beneficial effects of the technical solution provided by this invention are as follows: By acquiring multi-dimensional feature data of each storage fragment in the target storage space and determining the fragment type of each storage fragment based on the multi-dimensional feature data, the processing of storage fragments no longer relies solely on a single attribute, but can comprehensively reflect their overall usage characteristics in the cloud storage system. Furthermore, a target organization strategy matching the fragment type of each storage fragment is generated, and organization operations on each storage fragment in the target storage space are executed according to the generated target organization strategy. This enables the organization behavior to accurately adapt to the actual characteristics of different types of storage fragments, thereby transforming the traditional "one-size-fits-all" static organization that relies on a single dimension (such as object size) into a dynamic intelligent optimization process of "multi-dimensional identification, classification customization, and precise execution." This effectively solves the problem of poor organization results caused by the use of a single threshold organization strategy in the prior art, improves the organization effect of storage fragments, and thus improves the space utilization and overall operating performance of the cloud storage system. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a schematic diagram of an application environment for a storage fragmentation processing method based on a cloud storage system according to an embodiment of the present invention; Figure 2 This is a flowchart of a storage fragmentation processing method based on a cloud storage system according to an embodiment of the present invention; Figure 3 This is another flowchart of a storage fragmentation processing method based on a cloud storage system in one embodiment of the present invention; Figure 4 This is another flowchart of a storage fragmentation processing method based on a cloud storage system in one embodiment of the present invention; Figure 5 This is another flowchart of a storage fragmentation processing method based on a cloud storage system in one embodiment of the present invention; Figure 6 This is another flowchart of a storage fragmentation processing method based on a cloud storage system in one embodiment of the present invention; Figure 7 This is another flowchart of a storage fragmentation processing method based on a cloud storage system in one embodiment of the present invention; Figure 8 This is a schematic diagram of a storage fragmentation processing device based on a cloud storage system according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The storage fragmentation processing method based on a cloud storage system provided in this invention can be applied to, for example... Figure 1 The application environment shown. Specifically, this storage fragmentation processing method based on a cloud storage system is applied in a storage fragmentation processing system based on a cloud storage system, which includes, for example, […]. Figure 1 The diagram illustrates a client and server that communicate over a network to implement a storage fragmentation handling method based on a cloud storage system. The client, also known as the user terminal, is a program that provides local services to the client, corresponding to the server. Clients can include, but are not limited to, various personal computers, laptops, smartphones, and tablets. The server can be a standalone server or a server cluster consisting of multiple servers.

[0014] In one embodiment, such as Figure 2 As shown, a storage fragmentation processing method based on a cloud storage system is provided, which is then applied to... Figure 1 Taking the server in the example, the following steps are included: S101: Obtain multi-dimensional feature data of each storage fragment in the target storage space.

[0015] In this embodiment, it can be understood that the target storage space refers to a specific logical storage unit in the cloud storage system that requires fragmentation optimization. For example, the target storage space can be a specific storage bucket in the cloud storage system or a block storage volume. In actual business scenarios, its specific form could be a transaction log storage bucket in the financial field or a medical image archive volume in the medical field.

[0016] The cloud storage system may include, but is not limited to, systems based on object storage architecture (such as Amazon S3, Alibaba Cloud OSS, Tencent Cloud COS), systems based on block storage architecture (such as Amazon EBS, Alibaba Cloud Disk), systems based on file system storage architecture (such as Amazon EFS, Alibaba Cloud NAS), and hybrid storage architecture systems composed of one or more of the above storage services, without any limitation here.

[0017] Storage fragments (i.e., storage fragments) refer to storage fragments or data blocks that are not effectively utilized due to frequent writing, deletion, or updating. They include not only "small files" in the traditional sense, but also objects that are inefficient to store due to access patterns or lifecycles.

[0018] Multi-dimensional feature data refers to a set of features used to comprehensively characterize the state and business attributes of each storage fragment. Specifically, multi-dimensional feature data can include, but is not limited to: Physical dimension characteristics: such as the physical size and storage location information of storage fragments; Behavioral characteristics: such as the access frequency of storage fragments, the time of the most recent access, the creation time, and the last modification time; Semantic / business dimension features: For example, the data type identifier of storage fragments (such as log files, database backups, image files) and the tags assigned by users or business systems (such as "hot data", "compliance archives", "diagnostic graphs").

[0019] Specifically, multi-dimensional feature data may include, but is not limited to, at least two of the following: object size, access frequency, creation time or last modification time, data type, and business tags. For example, in a financial scenario, this may include the size of transaction logs, real-time access frequency, and compliant retention period tags; in a medical scenario, it may include the size of image files, clinical access frequency, and disease classification tags.

[0020] In one specific implementation, a data collector integrated into the cloud storage system can continuously or periodically scan the target storage space (such as an Amazon S3 bucket). The data collector extracts metadata and access logs for each storage fragment, thereby obtaining multi-dimensional feature data including but not limited to the following dimensions: physical dimension features, such as obtaining the physical size and storage location of the storage fragment by calling the metadata application programming interface (API) of the cloud storage service; behavioral dimension features, such as obtaining the access frequency, recent access time, and creation time of the storage fragment within a statistical period by querying an independent access log analysis service; and semantic dimension features, such as obtaining the data type identifier of the storage fragment and business tags (such as hot-data, compliance-archive) defined by users or operation and maintenance policies through the tag database of the associated business system.

[0021] The data acquisition unit aggregates, aligns, and formats the feature information from the discrete sources mentioned above, and finally generates a standardized multi-dimensional feature description vector (i.e., multi-dimensional feature data) for each storage fragment, thereby completing the acquisition of multi-dimensional feature data of each storage fragment in the target storage space.

[0022] S102: Based on the multi-dimensional feature data of each storage fragment, determine the fragment type to which each storage fragment belongs.

[0023] It is understood that in this embodiment, fragment type refers to the category assigned to each storage fragment based on the inherent attributes reflected by multi-dimensional feature data and through a predetermined classification logic. Fragment type may include, but is not limited to, high-frequency fragments during active periods, low-frequency fragments during active periods, and fragments during archiving periods.

[0024] Specifically, the cloud storage system includes a fragment classification decision module for executing step S102. This module receives a standardized multi-dimensional feature description vector generated for each storage fragment in step S101 as input, processes it based on predetermined classification logic, and outputs the fragment type determined for that storage fragment. The predetermined classification logic can be implemented in at least one of the following ways.

[0025] In one specific implementation, the predetermined classification logic includes a classification method based on a pre-configured rule set. The fragment classification decision module has multiple pre-set classification rules, each defining a mapping relationship between a combination of conditions based on one or more dimensions of feature data and a fragment type. For each storage fragment, the fragment classification decision module traverses the multiple classification rules, determining whether the multi-dimensional feature data of the storage fragment satisfies all the conditions defined by any rule. When the feature data of the storage fragment satisfies all the conditions of a certain rule, the fragment type mapped by that rule is determined as the fragment type of the storage fragment. This method, based on explicit rules, is characterized by clear logic, high execution efficiency, and ease of interpretation.

[0026] For example, in a storage optimization scenario of a financial transaction system, the fragment classification decision module pre-sets the following three classification rules: Rule A: If the "last access time" characteristic value of a certain storage fragment is within the last 7 days, and its "average daily access frequency" characteristic value is higher than 1000 times, and its business tag includes "real-time risk control", then the storage fragment is mapped as an "active period high-frequency fragment". Rule B: If a storage fragment's business tag contains "compliant archiving", or (its "last modified time" characteristic value is earlier than 365 days and its "access frequency" characteristic value is 0), then the storage fragment is mapped to an "archiving period fragment". Rule C: If the "last access time" characteristic value of a storage fragment is within the last 30 days, but its "average daily access frequency" characteristic value is less than 50 times, then the storage fragment is mapped as a "low-frequency fragment during the active period".

[0027] When processing a transaction log file, its multi-dimensional feature data is first obtained: the storage fragment size is 2MB, the last access time is "2x2x year xx month xx day", the average daily access frequency is "1200 times", and the business tag is "real-time risk control". The fragment classification decision module traverses the above rules: the storage fragment meets all the conditions of rule A (access time within 7 days, frequency higher than 1000 times, tag matching), so there is no need to continue matching other rules, and its fragment type is directly determined as "high-frequency fragment during active period".

[0028] Conversely, for a historical reconciliation file, its characteristic data shows: the object size is 5GB, the business tag is "compliant archiving," and the number of accesses in the past year is 0. When traversing the rules, this storage fragment first meets the condition of rule B (tag matching), and is therefore identified as an "archiving period fragment."

[0029] It should be noted that the aforementioned predefined classification logic, including classification based on pre-configured rule sets, is only one method. Predefined classification logic can also include classification based on pre-trained machine learning classification models; details can be found below. Figure 3 The explanation of steps S121-S122 will not be repeated here.

[0030] S103: Generate a target compaction strategy for each fragment type based on the fragment type to which each storage fragment belongs; S104: Based on the generated target compaction strategy, perform compaction operations on each storage fragment of the target storage space.

[0031] In one specific implementation, the fragment type labeled for each storage fragment is first read, and the fragment type is determined by step S102. Then, a target compaction strategy for each fragment type is generated by matching against a predefined and stored strategy template. Further, compaction operations are performed on each storage fragment within the target storage space according to the generated target compaction strategy. This embodiment generates and executes compaction strategies that precisely match the fragment type, allowing the compaction behavior to adapt to the actual characteristics of different types of storage fragments. This process transforms the traditional "one-size-fits-all" static compaction that relies on a single dimension (such as object size) into a dynamic intelligent optimization process of "multi-dimensional identification, classification customization, and precise execution," effectively solving the problem of poor compaction results caused by using a single threshold compaction strategy in existing technologies. This improves the compaction effect and targeting of storage fragments, thereby improving the storage space utilization and overall operating performance of the cloud storage system.

[0032] In summary, the beneficial effects of the technical solution provided by this invention are as follows: By acquiring multi-dimensional feature data of each storage fragment in the target storage space and determining the fragment type of each storage fragment based on the multi-dimensional feature data, the processing of storage fragments no longer relies solely on a single attribute, but can comprehensively reflect their overall usage characteristics in the cloud storage system. Furthermore, a target organization strategy matching the fragment type of each storage fragment is generated, and organization operations on each storage fragment in the target storage space are executed according to the generated target organization strategy. This enables the organization behavior to accurately adapt to the actual characteristics of different types of storage fragments, thereby transforming the traditional "one-size-fits-all" static organization that relies on a single dimension (such as object size) into a dynamic intelligent optimization process of "multi-dimensional identification, classification customization, and precise execution." This effectively solves the problem of poor organization results caused by the use of a single threshold organization strategy in the prior art, improves the organization effect of storage fragments, and thus improves the space utilization and overall operating performance of the cloud storage system.

[0033] In one embodiment, such as Figure 3As shown, step S102, which involves determining the fragment type of each storage fragment based on its multi-dimensional feature data, includes the following steps: S121: Input the multi-dimensional feature data of each storage fragment into a pre-trained machine learning classification model; S122: Determine the fragment type of each storage fragment based on the output of the machine learning classification model; The fragment types include at least those based on access frequency and / or those based on data lifecycle status. These fragment types may include, but are not limited to, high-frequency fragments during the active period (storage fragments accessed frequently recently), low-frequency fragments during the active period (storage fragments accessed less frequently recently), and archive fragments (historical storage fragments that have not been accessed for a long time), etc., without further limitation here.

[0034] In this embodiment, it can be understood that a pre-trained machine learning classification model refers to a computational model that is pre-trained using a machine learning algorithm and is capable of automatic classification and prediction. The training process of this model specifically involves: using historically accumulated storage fragment samples whose fragment types have been accurately labeled by operational rules or expert experience, taking their multi-dimensional feature data as input features, and their labeled fragment types as target labels, and learning the complex mapping relationship from the multi-dimensional feature space to the fragment type space through an algorithm.

[0035] In another specific implementation, the fragment classification decision module integrates a pre-trained machine learning classification model. This model is trained on historical multi-dimensional feature data labeled with fragment types, enabling it to learn and establish a mapping relationship from the feature space to the fragment type space. The fragment classification decision module inputs the multi-dimensional feature data of each stored fragment into the machine learning classification model and determines the fragment type with the highest probability based on the probability distribution output by the model (i.e., the classification result), thus classifying it as the fragment type of the stored fragment. This approach can automatically capture the complex non-linear relationship between features and types, exhibiting stronger adaptability and classification accuracy.

[0036] In one embodiment, such as Figure 4 As shown, in step S103, the fragment types include high-frequency fragments during the active period, low-frequency fragments during the active period, and fragments during the archiving period. That is, the process of generating a target compaction strategy for each fragment type, based on the fragment type of each storage fragment, includes the following steps: S131: If the fragment type is a high-frequency fragment during the active period, a first reorganization strategy is generated, which mainly focuses on index reorganization and / or proximity merging. The first reorganization strategy aims to minimize access latency. S132: If the fragment type is a low-frequency fragment during the active period or a fragment during the archive period, a second compaction strategy is generated, which is based on space compression and merging. The second compaction strategy aims to maximize storage space utilization.

[0037] In this embodiment, it can be understood that the target defragmentation strategy refers to a storage defragmentation operation scheme formulated according to the fragmentation type, which may include, but is not limited to, the first defragmentation strategy and the second defragmentation strategy.

[0038] The first consolidation strategy is designed for storage fragments that are high-frequency fragments during their active period. This strategy focuses on index reorganization and / or proximity merging as core operations, aiming to minimize access latency by optimizing the metadata index structure and physical storage layout of the storage fragments.

[0039] The second consolidation strategy targets storage fragments that are infrequently occurring during periods of high activity or during periods of archiving. This strategy focuses on space compression and merging as its core operation, aiming to maximize storage space utilization by reducing storage redundancy and consolidating fragmented spaces.

[0040] In one specific implementation, a corresponding target compaction strategy is generated based on the fragment type to which each storage fragment belongs: for storage fragments whose fragment type is determined to be a high-frequency fragment during the active period, a first compaction strategy is generated, which mainly focuses on index reorganization (e.g., optimizing the tree structure of its metadata index) and / or proximity merging (e.g., adjusting logically related or frequently accessed data blocks to adjacent positions in physical storage) to ensure high access performance while reducing the generation of new fragments.

[0041] For storage fragments identified as low-frequency fragments during active periods or archive periods, a second compaction strategy is generated, which primarily involves space compression and merging across storage units (e.g., using a high compression ratio algorithm to batch compress multiple such objects and merge them into larger data units or migrate them to a dedicated archive storage layer) to maximize storage space utilization and reduce storage costs.

[0042] The effect of this embodiment is that by formulating differentiated organization strategies for different fragment types, it can simultaneously take into account the access efficiency of high-frequency data and the storage space optimization of low-frequency data, thereby achieving a comprehensive improvement in the efficient operation and resource utilization of the cloud storage system.

[0043] For example, in a financial transaction system, when frequently accessed transaction log fragments are identified as high-frequency fragments during the active period, a first consolidation strategy of index reorganization and proximity merging is adopted to ensure that transaction query and real-time analysis performance are not affected. For archive fragments that store medical image data that has not been accessed for the past three years, a second consolidation strategy of space compression and merging across storage units is adopted to consolidate the fragments into the archive storage area, freeing up storage space and reducing storage costs. For MRI image data that has been accessed occasionally in the past six months as low-frequency fragments during the active period, the second consolidation strategy is also applied to achieve space optimization while retaining the necessary access performance.

[0044] In one embodiment, such as Figure 5 As shown, in step S103, after generating the target compaction strategy for each fragment type based on the fragment type of each storage fragment, the following steps are also included: S103A1: Obtain the operating status parameters of the cloud storage system; S103A2: Based on the operating status parameters, set resource consumption constraints for the first and / or second cleaning strategies; The resource consumption constraints include at least one of the following: peak CPU utilization, network bandwidth limit, and storage I / O quota. The constraint value of the resource consumption constraints is dynamically determined based on the system resource reserve reflected by the operating status parameters.

[0045] In this embodiment, it can be understood that: operating status parameters refer to a set of real-time or near-real-time indicators used to reflect the current resource usage and availability status of the cloud storage system, which is an instantaneous snapshot of the system status. Examples include central processing unit (CPU) utilization, memory usage, network input / output bandwidth utilization, and storage device input / output operation (I / O) latency and throughput.

[0046] Resource consumption constraints refer to boundary conditions set to limit the system resource consumption of the sorting task, including but not limited to peak CPU utilization, network bandwidth limit and storage I / O quota.

[0047] System resource reserves refer to the amount of remaining resources in the cloud storage system that can be used by the organization task in the current running state. It is usually obtained by subtracting the current actual use from the total resources.

[0048] Dynamic determination means that the constraint values ​​are not fixed configurations, but rather calculated and distributed in real time or periodically based on the operating status parameters of the cloud storage system and the remaining system resources.

[0049] In one specific implementation, after generating the target consolidation strategy, the operating status parameters of the cloud storage system are acquired, including the current CPU utilization, network bandwidth utilization, and storage I / O utilization. Based on these operating status parameters, the system resource reserve is calculated, and resource consumption constraints are set for the first and / or second consolidation strategies accordingly. These constraints include at least one of peak CPU utilization, network bandwidth limit, and storage I / O quota, with the constraint values ​​determined based on the resource reserve. This approach prevents consolidation operations from impacting online service performance under high load and improves consolidation efficiency when resources are abundant, thereby optimizing the overall consolidation effect while ensuring service SLAs.

[0050] In one embodiment, such as Figure 8 As shown, it also includes the following steps: S103B1: Obtain system load prediction information. The system load prediction information is generated based on the historical operating data of the cloud storage system to predict the load situation for a preset future time period. S103B2: Based on the operating status parameters and system load prediction information, set the execution time window and / or execution priority for the first and / or second cleaning strategies.

[0051] In this embodiment, it can be understood that: it refers to the structured data output by training a prediction model using historical operating data of the cloud storage system to predict the trend of key performance indicators such as central processing unit (CPU) utilization, network bandwidth utilization, and storage input / output (I / O) operation frequency within a preset future time period.

[0052] The execution time window refers to the allowed scheduling period allocated to tasks based on load forecasting and real-time status, used to control task execution within the time interval allowed by system load. Execution priority refers to the scheduling weight dynamically calculated based on task type, resource requirements, and system load status in multi-task scenarios.

[0053] In one specific implementation, after acquiring the operating status parameters, a step of acquiring system load prediction information is also performed. The system load prediction information and the operating status parameters are used as joint inputs, and based on these, execution time windows and / or execution priorities are dynamically set for the first and / or second defragmentation strategies. Specifically, based on the current resource availability reflected in the operating status parameters and the future load trend revealed by the system load prediction information, suitable low-load periods for executing the first and second defragmentation strategies are calculated and used as execution time windows, or different execution priorities are assigned to different strategies under resource contention. This allows the execution of the first and second defragmentation strategies to obtain a larger execution window and quickly reclaim fragmentation when resources are sufficient, while also completing or downgrading defragmentation tasks before the expected load peak arrives, thereby maximizing defragmentation efficiency and resource utilization while ensuring the Service Level Agreement (SLA) for online services.

[0054] For example, in a financial institution's cloud storage system, system load prediction information is generated by a Long Short-Term Memory (LSTM) network model. The prediction shows that 09:00-11:00 and 14:00-16:00 the next day are high-load periods, while 00:00-05:00 at night is a period of sustained low load. Operating status parameters show that the current CPU utilization is 75% and the network bandwidth utilization is 60%. Based on this, in step S103B2, the execution time window for the second consolidation strategy (space compression and merging) is set to 00:00-05:00 the next day, and a higher execution priority is assigned to ensure its completion within this window. Simultaneously, multiple shorter execution time windows are set for the first consolidation strategy (index reorganization) (e.g., 02:00-02:30 and 03:30-04:00 daily), and dynamic priorities are configured so that it can be prioritized when the real-time operating status parameters indicate a low instantaneous system load, and its priority automatically decreases when resources are scarce.

[0055] In one embodiment, prior to step S103A2, that is, before setting resource consumption constraints for the first and / or second tidying strategies, the following steps are further included: S103C1: Based on a pre-built resource consumption prediction model, estimate the amount of computing and network resources required to execute the first and / or second tidying strategies; The dynamic determination of the constraint value of resource consumption constraints also combines the estimated amount of computing resources and network resources.

[0056] In this embodiment, it can be understood that the pre-built resource consumption prediction model refers to a computational model that is constructed and validated before system deployment, based on historical data from data processing tasks or offline performance benchmark results, through statistical analysis or machine learning methods. This model is used to predict the amount of system resources that may be consumed when executing a specific data processing strategy, based on input task characteristic parameters (such as the total amount of data to be processed, operation type, etc.).

[0057] Computing resource quantity refers to the quantitative estimate of computing resources such as central processing unit (CPU) operation cycles, memory usage, and disk input / output (I / O) operations required to execute the first and / or second cleaning strategies.

[0058] Network resource quantity refers to a quantitative estimate of the total network bandwidth usage or data transmission volume required to execute the first and / or second cleanup strategies.

[0059] In one specific implementation, before setting the final resource consumption constraints for the defragmentation strategy, step S103C is first executed: a pre-built resource consumption prediction model is invoked, taking the task characteristics of the defragmentation strategy to be executed (such as associated fragment types, total data volume of storage fragments to be defragmented, average object size, planned operation type, etc.) as input. Based on this, the model outputs an estimated value of the computational and network resources required for the task. Subsequently, when setting resource consumption constraints (step S103A2), this estimated value is jointly analyzed and weighed with the current system resource reserves reflected by real-time acquired operating status parameters, thereby calculating more accurate and safer constraint values ​​(such as peak CPU utilization, network bandwidth limit, etc.). Thus, by introducing quantitative estimation based on a pre-built resource consumption prediction model during the constraint setting stage, resource consumption constraints can more accurately reflect the actual resource requirements of the defragmentation task, thereby reducing resource waste or business interference caused by insufficient estimation or excessive conservatism, and improving the controllability and execution efficiency of fragment defragmentation.

[0060] For example, in a financial institution's cloud storage system, a second defragmentation strategy (cross-storage unit space compression and merging) is planned to be executed on a batch of identified archived fragments. Before execution, a resource consumption prediction model is first invoked, with the scale parameters of this task input (e.g., the total number of storage fragments to be merged is 100,000, and the average size is 100MB). Based on historical task data, the model calculates an estimated value: completing this batch defragmentation is expected to consume computing resources equivalent to 4 CPU cores running continuously for 2 hours, and average network resources with a peak of approximately 200 Mbps.

[0061] Subsequently, based on the real-time monitored operating status parameters (e.g., the current cluster has approximately 3 CPU cores available and 500 Mbps of available network bandwidth), a comprehensive decision is made to dynamically set the resource consumption constraints for this task: the peak CPU utilization constraint is set to "allowing a maximum of 3 CPU cores to be used instantaneously" (this value is based on the predicted value and takes into account the current available capacity, satisfying the main task requirements while reserving a buffer for business needs), the network bandwidth limit is set to "not exceeding 200 Mbps" (consistent with the predicted peak), and the storage I / O quota is set to not exceed the specified IOPS threshold per day.

[0062] In one embodiment, such as Figure 7 As shown, in step S104, which involves performing the defragmentation operation on each storage fragment of the target storage space according to the generated target defragmentation strategy, the following steps are included: S141: Based on the execution time window and / or execution priority, trigger a compaction task based on the first compaction strategy and / or the second compaction strategy to perform compaction operations on each storage fragment of the target storage space; S142: During the execution of the sorting task, monitor the actual resource usage of the cloud storage system based on the resource consumption constraints set for each sorting strategy; S143: If the actual resource usage exceeds the associated resource consumption constraints, the execution rate of the corresponding task will be dynamically adjusted or suspended based on the execution priority.

[0063] In one specific implementation, firstly, based on the execution time window and execution priority pre-set for the defragmentation task, the scheduler triggers a specific defragmentation task based on a first defragmentation strategy or a second defragmentation strategy when the time and resource conditions are met, and performs defragmentation operations on each storage fragment of the target storage space. Subsequently, during the execution of the defragmentation task, the actual resource usage of the cloud storage system is continuously collected and compared according to the resource consumption constraints set for each defragmentation strategy (such as peak CPU utilization and network bandwidth limit) (S142). If the actual resource usage of the system exceeds the resource consumption constraints associated with the currently executing task, intervention is immediately implemented based on the execution priority of the task: for tasks with lower execution priority, their execution rate is dynamically adjusted by reducing their data processing throughput; if the resource limit is severely exceeded or resources need to be guaranteed for higher priority tasks (including online services), an instruction to suspend the execution of the corresponding defragmentation task is issued (S143). This embodiment achieves fine-grained control over background defragmentation operations, ensuring that they operate within the preset resource safety boundaries, thereby effectively reclaiming storage fragments while prioritizing the performance stability of core online services.

[0064] For example, in a financial institution's cloud storage system, according to the scheduling arrangement of the target consolidation strategy, the execution time window for the cross-storage unit compression and merging task for historical reconciliation files (second consolidation strategy) is set to 00:00-05:00 at night, and an execution priority of 5 is assigned (the larger the value, the lower the priority); at the same time, the execution time window for the index reorganization task for recent transaction logs (first consolidation strategy) is set to 02:00-02:30, and an execution priority of 1 is assigned (the smaller the value, the higher the priority).

[0065] After the task is initiated, real-time monitoring is performed based on the resource consumption constraints set for each defragmentation strategy (e.g., second defragmentation strategy: network bandwidth limit 200 Mbps, CPU peak utilization ≤10%; first defragmentation strategy: CPU peak utilization ≤20%, network bandwidth limit 100 Mbps). If the cluster CPU utilization is found to surge from 60% to 85% during monitoring, exceeding the CPU peak utilization constraint of the second defragmentation strategy, the execution rate of the second defragmentation strategy task with an execution priority value of 5 (lower) is first limited (e.g., its network bandwidth limit is dynamically reduced from 200 Mbps to 80 Mbps). If the resource shortage continues to worsen, and resources need to be reserved for the first defragmentation strategy with an execution priority value of 1 (higher) or sudden online business, some or all second defragmentation strategy tasks may be suspended until the running status parameters return to a safe level before resuming. In this way, defragmentation can be effectively promoted within a predetermined time window, while intelligent adjustments can be made during sudden resource shortages to ensure that the processing performance of critical transactions is not affected.

[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] In one embodiment, a storage fragmentation processing device based on a cloud storage system is provided, which corresponds one-to-one with the storage fragmentation processing method based on a cloud storage system described in the above embodiments. For example... Figure 8 As shown, the storage fragmentation processing device based on the cloud storage system includes a first acquisition module 201, a determination module 202, a generation module 203, and an execution module 204. Detailed descriptions of each functional module are as follows: The first acquisition module 201 is used to acquire multi-dimensional feature data of each storage fragment in the target storage space; The determining module 202 is used to determine the fragment type of each storage fragment based on the multi-dimensional feature data of each storage fragment; The generation module 203 is used to generate a target compaction strategy for each of the storage fragments based on the fragment type to which each of the storage fragments belongs; The execution module 204 is used to perform a defragmentation operation on each storage fragment of the target storage space according to the generated target defragmentation strategy.

[0068] In one embodiment, the determining module 202 is further configured to: The multi-dimensional feature data of each storage fragment are input into a pre-trained machine learning classification model; Based on the output of the machine learning classification model, determine the fragment type to which each storage fragment belongs; The fragment types include at least those based on access frequency and / or those based on data lifecycle status.

[0069] In one embodiment, the fragment types include high-frequency fragments during the active period, low-frequency fragments during the active period, and fragments during the archiving period; the generation module 203 is further configured to: If the fragment type is the high-frequency fragment during the active period, a first reorganization strategy is generated, which is based on index reorganization and / or proximity merging. The first reorganization strategy aims to minimize access latency. If the fragment type is the active period low-frequency fragment or the archive period fragment, a second sorting strategy based on space compression and merging is generated, with the goal of maximizing storage space utilization.

[0070] In one embodiment, the storage fragmentation processing device further includes: The second acquisition module is used to acquire the operating status parameters of the cloud storage system; The first setting module is used to set resource consumption constraints for the first and / or second cleaning strategies based on the running status parameters. The resource consumption constraints include at least one of peak CPU utilization, network bandwidth limit, and storage I / O quota, and the constraint value of the resource consumption constraints is dynamically determined based on the system resource reserve reflected by the operating status parameters.

[0071] In one embodiment, the storage fragmentation processing device further includes: The third acquisition module is used to acquire system load prediction information, which is generated based on the historical operating data of the cloud storage system to predict the load situation for a preset future time period. The second setting module is used to set the execution time window and / or execution priority for the first and / or the second cleaning strategy based on the running status parameters and the system load prediction information.

[0072] In one embodiment, the storage fragmentation processing device further includes: The estimation module is used to estimate the amount of computing and network resources required to execute the first and / or the second consolidation strategies based on a pre-built resource consumption prediction model. The dynamic determination of the constraint value of the resource consumption constraint also combines the estimated amount of computing resources and the amount of network resources.

[0073] In one embodiment, the execution module 204 is further configured to: Based on the execution time window and / or the execution priority, a defragmentation task based on the first defragmentation strategy and / or the second defragmentation strategy is triggered to perform defragmentation operations on each storage fragment of the target storage space. During the execution of the sorting task, the actual resource usage of the cloud storage system is monitored based on the resource consumption constraints set for each sorting strategy. If the actual resource usage is detected to exceed the associated resource consumption constraint, the execution rate of the corresponding task will be dynamically adjusted or the task will be paused based on the execution priority.

[0074] Specific limitations regarding the storage fragmentation processing device based on cloud storage systems can be found in the limitations of the storage fragmentation processing method based on cloud storage systems mentioned above, and will not be repeated here. Each module in the aforementioned storage fragmentation processing device based on cloud storage systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0075] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for implementing a storage fragmentation processing method based on a cloud storage system. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a storage fragmentation processing method based on a cloud storage system.

[0076] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the storage fragmentation processing method based on a cloud storage system described in the above embodiments, for example... Figure 2 The steps S101-S104 and their associated steps are shown below, or Figures 2 to 7 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes a computer program, it implements the functions of each module / unit in this embodiment of the storage fragmentation processing device based on a cloud storage system, for example... Figure 8 The functions of the storage fragmentation processing device based on the cloud storage system shown are not described again here to avoid repetition.

[0077] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the storage fragmentation processing method based on a cloud storage system described in the above embodiments, for example... Figure 2 The steps S101-S104 and their associated steps are shown below, or Figures 2 to 7 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the storage fragmentation processing device based on the cloud storage system, for example... Figure 8 The functions of the storage fragmentation processing device based on the cloud storage system shown are not described again here to avoid repetition. The computer-readable storage medium can be non-volatile or volatile.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0080] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A storage fragmentation processing method based on a cloud storage system, characterized in that, include: Obtain multi-dimensional feature data of each storage fragment in the target storage space; Based on the multi-dimensional feature data of each storage fragment, the fragment type of each storage fragment is determined; Based on the fragment type to which each of the storage fragments belongs, generate a target compaction strategy for each of the fragment types; Based on the generated target defragmentation strategy, defragmentation operations are performed on each storage fragment of the target storage space.

2. The method according to claim 1, characterized in that, The determination of the fragment type of each storage fragment based on its multi-dimensional feature data includes: The multi-dimensional feature data of each storage fragment are input into a pre-trained machine learning classification model; Based on the output of the machine learning classification model, determine the fragment type to which each storage fragment belongs; The fragment types include at least those based on access frequency and / or those based on data lifecycle status.

3. The method according to claim 1 or 2, characterized in that, The fragment types include high-frequency fragments during the active period, low-frequency fragments during the active period, and fragments during the archive period; The step of generating a target compaction strategy for each of the storage fragment types includes: If the fragment type is the high-frequency fragment during the active period, a first reorganization strategy is generated, which is based on index reorganization and / or proximity merging. The first reorganization strategy aims to minimize access latency. If the fragment type is the active period low-frequency fragment or the archive period fragment, a second sorting strategy based on space compression and merging is generated, with the goal of maximizing storage space utilization.

4. The method according to claim 3, characterized in that, After generating a target compaction strategy for each of the storage fragment types based on the fragment type to which each of the storage fragments belongs, the method further includes: Obtain the operating status parameters of the cloud storage system; Based on the operating status parameters, resource consumption constraints are set for the first and / or second cleaning strategies. The resource consumption constraints include at least one of peak CPU utilization, network bandwidth limit, and storage I / O quota, and the constraint value of the resource consumption constraints is dynamically determined based on the system resource reserve reflected by the operating status parameters.

5. The method according to claim 4, characterized in that, Also includes: Obtain system load prediction information, which is generated based on the historical operating data of the cloud storage system to predict the load situation for a preset future time period; Based on the operating status parameters and the system load prediction information, set the execution time window and / or execution priority for the first and / or second consolidation strategies.

6. The method according to claim 5, characterized in that, Before setting resource consumption constraints for the first and / or second cleanup strategies, the method further includes: Based on a pre-built resource consumption prediction model, the amount of computing and network resources required to execute the first and / or the second tidying strategy is estimated. The dynamic determination of the constraint value of the resource consumption constraint also combines the estimated amount of computing resources and the amount of network resources.

7. The method according to claim 5 or 6, characterized in that, The step of performing defragmentation operations on each storage fragment of the target storage space according to the generated target defragmentation strategy includes: Based on the execution time window and / or the execution priority, a defragmentation task based on the first defragmentation strategy and / or the second defragmentation strategy is triggered to perform defragmentation operations on each storage fragment of the target storage space. During the execution of the sorting task, the actual resource usage of the cloud storage system is monitored based on the resource consumption constraints set for each sorting strategy. If the actual resource usage exceeds the associated resource consumption constraint, the execution rate of the corresponding task will be dynamically adjusted or suspended based on the execution priority.

8. A storage fragmentation processing device based on a cloud storage system, characterized in that, include: The first acquisition module is used to acquire multi-dimensional feature data of each storage fragment in the target storage space; The determination module is used to determine the fragment type of each storage fragment based on the multi-dimensional feature data of each storage fragment; The generation module is used to generate a target compaction strategy for each of the storage fragments based on the fragment type to which each of the storage fragments belongs; The execution module is used to perform reorganization operations on each storage fragment of the target storage space according to the generated target reorganization strategy.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the storage fragmentation processing method based on the cloud storage system as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the storage fragmentation processing method based on the cloud storage system as described in any one of claims 1 to 7.

Images