A data cleaning method and device, electronic equipment and storage medium

Abstract

This invention relates to the field of computer technology and discloses a data cleaning method, apparatus, electronic device, and storage medium. The method includes: acquiring data characteristic information, data type, and current business mode of each data item in a storage system; determining the access mode of any data item based on its data characteristic information and data type; determining a target data cleaning strategy for the storage system based on the current business mode and the access modes of each data item; and cleaning the storage system according to the target data cleaning strategy. The method provided above improves both the efficiency and reliability of data cleaning results by predictively analyzing the access modes of data items and combining this with the current business mode of the storage system to determine the target data cleaning strategy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data cleaning method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, with the rapid development of technologies such as big data and cloud computing, storage systems, especially centralized storage systems, are under tremendous pressure. After long-term use, storage systems accumulate a large amount of redundant and useless data, resulting in a waste of storage resources. Therefore, how to clean up data in storage systems has become a key research topic.

[0003] In related technologies, manual operation is usually relied upon, with technicians using their experience to filter redundant and useless data in the storage system and perform data cleanup to free up storage space in the storage system.

[0004] However, due to the cumbersome manual operation process and the difficulty for technicians to accurately screen the data to be cleaned, there may be cases of data being cleaned incorrectly or data not being screened completely. This not only reduces the efficiency of data cleaning but also fails to guarantee the reliability of the data cleaning results. Summary of the Invention

[0005] This application provides a data cleaning method, apparatus, electronic device, and storage medium to address the shortcomings of related technologies, such as reduced data cleaning efficiency and inability to guarantee the reliability of data cleaning results.

[0006] The first aspect of this application provides a data cleaning method, including:

[0007] Obtain the data characteristic information of each data item in the storage system, the data type of each data item, and the current business mode of the storage system;

[0008] For any of the data items, the access mode of the data item is determined based on the data feature information and data type of the data item;

[0009] Based on the current business mode of the storage system and the access mode of each data item, determine the target data cleanup strategy for the storage system;

[0010] Data cleaning is performed on the storage system according to the target data cleaning strategy of the storage system.

[0011] In one optional implementation, determining the access mode of any data item based on its data characteristic information and data type includes:

[0012] For any of the data items, determine the access pattern analysis strategy for the data item based on the data item's data type;

[0013] According to the access pattern analysis strategy of the data item, the access pattern of the data item is analyzed based on the data feature information of the data item;

[0014] The data types mentioned above are at least divided into two types: structured data and unstructured data.

[0015] In one optional implementation, determining the access pattern analysis strategy for any data item based on its data type includes:

[0016] For any of the data items, if the data item is structured data, an access pattern analysis strategy for the data item is determined based on the detailed type of the data item characterized by the data feature information of the data item.

[0017] In the case that the data item is unstructured data, an access pattern analysis strategy for the data item is determined based on the data format represented by the data feature information of the data item.

[0018] The detailed types of the data items are at least divided into two types: order data items and user information data items. The data formats include image formats, video formats, and audio formats.

[0019] In one optional implementation, the step of analyzing the access pattern of the data item according to the access pattern analysis strategy of the data item, based on the data feature information of the data item, includes:

[0020] For any of the data items, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, the access frequency of the data item in a preset future period is predicted, and the future access frequency prediction result of the data item is obtained.

[0021] The access pattern of the data item is determined based on the predicted future access frequency of the data item.

[0022] In one optional implementation, determining the target data cleanup strategy for the storage system based on the current business mode of the storage system and the access mode of each data item includes:

[0023] Based on the access patterns of each data item, low-frequency access data items are filtered in the storage system;

[0024] Analyze the access cycle information of each of the low-frequency access data items to determine the last access time of each of the low-frequency access data items;

[0025] Based on the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items.

[0026] When the current business mode of the storage system is normal mode, the short-term low-frequency access data items are designated as data items to be deleted, and the long-term low-frequency access data items are designated as data items to be migrated.

[0027] The target data cleanup strategy for the storage system is determined to be to delete the data items to be deleted and to migrate the data items to be migrated to the storage medium.

[0028] When the current business mode of the storage system is high availability mode, the short-term low-frequency access data items are regarded as data items to be migrated, and the long-term low-frequency access data items are regarded as data items to be compressed.

[0029] The target data cleanup strategy for the storage system is determined to be to perform storage medium migration processing on the data items to be migrated and data compression processing on the data items to be compressed.

[0030] The last access time of the short-term low-frequency access data item is later than the last access time of the long-term low-frequency access data item.

[0031] In an optional implementation, the method further includes:

[0032] Before deleting the data item to be deleted, a snapshot of the data item to be deleted is generated;

[0033] In the event of accidental deletion of data items, the data items can be recovered based on the snapshot data.

[0034] In an optional implementation, the method further includes:

[0035] Obtain the actual operating status and storage space usage of the storage system;

[0036] Based on the actual operation of the storage system and the usage of storage space, determine the data cleaning and optimization strategy;

[0037] Based on the data cleaning and optimization strategy, optimize the target data cleaning strategy.

[0038] A second aspect of this application provides a data cleaning apparatus, comprising:

[0039] The acquisition module is used to acquire data feature information of each data item in the storage system, the data type of each data item, and the current business mode of the storage system;

[0040] The first determining module is used to determine the access mode of any data item based on its data feature information and data type.

[0041] The second determining module is used to determine the target data cleaning strategy of the storage system based on the current business mode of the storage system and the access mode of each data item.

[0042] The cleaning module is used to clean the data in the storage system according to the target data cleaning strategy of the storage system.

[0043] In one optional implementation, the first determining module is specifically used for:

[0044] For any of the data items, determine the access pattern analysis strategy for the data item based on the data item's data type;

[0045] According to the access pattern analysis strategy of the data item, the access pattern of the data item is analyzed based on the data feature information of the data item;

[0046] The data types mentioned above are at least divided into two types: structured data and unstructured data.

[0047] In one optional implementation, the first determining module is specifically used for:

[0048] For any of the data items, if the data item is structured data, an access pattern analysis strategy for the data item is determined based on the detailed type of the data item characterized by the data feature information of the data item.

[0049] In the case that the data item is unstructured data, an access pattern analysis strategy for the data item is determined based on the data format represented by the data feature information of the data item.

[0050] The detailed types of the data items are at least divided into two types: order data items and user information data items. The data formats include image formats, video formats, and audio formats.

[0051] In one optional implementation, the first determining module is specifically used for:

[0052] For any of the data items, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, the access frequency of the data item in a preset future period is predicted, and the future access frequency prediction result of the data item is obtained.

[0053] The access pattern of the data item is determined based on the predicted future access frequency of the data item.

[0054] In one optional implementation, the second determining module is specifically used for:

[0055] Based on the access patterns of each data item, low-frequency access data items are filtered in the storage system;

[0056] Analyze the access cycle information of each of the low-frequency access data items to determine the last access time of each of the low-frequency access data items;

[0057] Based on the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items.

[0058] When the current business mode of the storage system is normal mode, the short-term low-frequency access data items are designated as data items to be deleted, and the long-term low-frequency access data items are designated as data items to be migrated.

[0059] The target data cleanup strategy for the storage system is determined to be to delete the data items to be deleted and to migrate the data items to be migrated to the storage medium.

[0060] When the current business mode of the storage system is high availability mode, the short-term low-frequency access data items are regarded as data items to be migrated, and the long-term low-frequency access data items are regarded as data items to be compressed.

[0061] The target data cleanup strategy for the storage system is determined to be to perform storage medium migration processing on the data items to be migrated and data compression processing on the data items to be compressed.

[0062] The last access time of the short-term low-frequency access data item is later than the last access time of the long-term low-frequency access data item.

[0063] In an optional implementation, the second determining module is further configured to:

[0064] Before deleting the data item to be deleted, a snapshot of the data item to be deleted is generated;

[0065] In the event of accidental deletion of data items, the data items can be recovered based on the snapshot data.

[0066] In one alternative embodiment, the device further includes:

[0067] An optimization module is used to obtain the actual operating status and storage space usage of the storage system; determine a data cleaning and optimization strategy based on the actual operating status and storage space usage of the storage system; and optimize the target data cleaning strategy based on the data cleaning and optimization strategy.

[0068] A third aspect of this application provides an electronic device, comprising: at least one processor and a memory;

[0069] The memory stores computer-executed instructions;

[0070] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.

[0071] The fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described in the first aspect above and various possible designs of the first aspect.

[0072] The fifth aspect of this application provides a computer program product including computer instructions for causing a computer to perform the methods described in the first aspect above and various possible designs of the first aspect.

[0073] The technical solution of this application has the following advantages:

[0074] This application provides a data cleaning method, apparatus, electronic device, and storage medium. The method includes: acquiring data characteristic information, data type, and current business mode of each data item in a storage system; determining the access mode of any data item based on its data characteristic information and data type; determining a target data cleaning strategy for the storage system based on the current business mode and access modes of each data item; and cleaning the storage system according to the target data cleaning strategy. The method provided above improves both data cleaning efficiency and reliability by predictively analyzing access modes of data items and combining this with the current business mode of the storage system to determine the target data cleaning strategy. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0076] Figure 1 This is a schematic diagram of the structure of the data cleaning system on which the embodiments of this application are based;

[0077] Figure 2 A flowchart illustrating the data cleaning method provided in this application embodiment;

[0078] Figure 3 This is a schematic diagram of the structure of a data cleaning device provided in an embodiment of this application;

[0079] Figure 4 This is a schematic diagram of another data cleaning device provided in an embodiment of this application;

[0080] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0081] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. In the following descriptions of embodiments, "a plurality of" means two or more, unless otherwise explicitly defined.

[0084] In related technologies, the rapid development of information technology inevitably leads to an increasing demand for the generation and storage of various types of data. The development of big data and cloud computing, in particular, puts increasing pressure on storage systems, especially high-end centralized storage systems. Whether structured or unstructured, data accumulates a large amount of redundant and useless data over time. This useless data not only occupies valuable storage space but also increases system maintenance costs and reduces overall storage efficiency. Traditional data cleaning methods typically rely on manual operations or simple automated scripts, which suffer from inefficiency, error-proneness, and an inability to quickly adapt to different data types and access patterns. Furthermore, data cannot be recovered promptly after errors occur. Therefore, there is an urgent need for an efficient, intelligent, and adaptable data cleaning mechanism to improve storage system utilization and reduce maintenance costs. Currently, there is no independent module in the industry to uniformly manage such an intelligent data cleaning mechanism. In some extreme scenarios, this can easily lead to inefficient cluster response to front-end services, thereby affecting the availability of the entire system.

[0085] To address the aforementioned problems, embodiments of this application provide a data cleaning method, apparatus, electronic device, and storage medium. The method includes: acquiring data characteristic information, data type, and current business mode of each data item in a storage system; for any given data item, determining its access mode based on its data characteristic information and data type; determining a target data cleaning strategy for the storage system based on its current business mode and the access modes of each data item; and cleaning the storage system according to the target data cleaning strategy. The method provided above improves both the efficiency and reliability of data cleaning results by predictively analyzing the access modes of data items and combining this with the current business mode of the storage system to determine the target data cleaning strategy.

[0086] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0087] First, the structure of the data cleaning system on which this application is based will be described:

[0088] The data cleaning method, apparatus, electronic device, and storage medium provided in this application are applicable to filtering data items to be cleaned in a storage system and performing corresponding cleaning processing on the data items to be cleaned. Figure 1The diagram shown is a structural schematic of the data cleaning system based on the embodiments of this application. It mainly includes a data acquisition device, a data storage system, and a data cleaning device. The data acquisition device collects information such as the data characteristics of each data item in the storage system, the data type of each data item, and the current business mode of the storage system. It then sends the collected information to the data cleaning device. Based on the obtained information, the data cleaning device formulates a target data cleaning strategy for the storage system and performs data cleaning according to the target data cleaning strategy to release the storage space of the storage system.

[0089] This application provides a data cleaning method for filtering data items to be cleaned in a storage system and performing corresponding cleaning processing on these data items. The execution subject of this application embodiment is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic devices that can be used for data cleaning.

[0090] like Figure 2 The diagram shown is a flowchart illustrating a data cleaning method provided in an embodiment of this application. The method includes:

[0091] Step 201: Obtain the data characteristic information of each data item in the storage system, the data type of each data item, and the current business mode of the storage system.

[0092] Specifically, data items in the storage system can be monitored in real time to record the data characteristics of each data item in the storage system, while also collecting and recording the data type of the data items and the current business mode of the storage system.

[0093] Step 202: For any data item, determine the access mode of the data item based on its data characteristics and data type.

[0094] The data characteristics of the data items include current access frequency, creation time, and modification time, and the data types are at least divided into structured data and unstructured data.

[0095] Specifically, by analyzing the data characteristics and data types of data items, we can predict the future access frequency and other access pattern characteristics of data items, thereby obtaining the access patterns of data items.

[0096] Step 203: Determine the target data cleanup strategy for the storage system based on the current business model of the storage system and the access patterns of each data item.

[0097] Specifically, the data items to be cleaned can be located in the storage system based on the current business model of the storage system and the access patterns of each data item, and then a specific target data cleanup strategy can be determined based on the data items to be cleaned.

[0098] The target data cleaning strategy includes the data cleaning methods for the data items to be cleaned.

[0099] Step 204: Clean the data in the storage system according to the target data cleaning strategy of the storage system.

[0100] Specifically, the corresponding data cleaning operations can be performed on the data items to be cleaned according to the data cleaning method characterized by the target data cleaning strategy, so as to clean up the data of the storage system and release the storage space of the storage system.

[0101] Specifically, in one embodiment, in order to further free up storage space in the storage system, duplicate data copies in the storage system can also be cleaned up to remove duplicate data in the storage system, reduce storage space waste, and improve space utilization.

[0102] Based on the above embodiments, since the storage system stores a large number of data items, and these data items exhibit a certain degree of diversity, to ensure the accuracy of the analysis and prediction results of data item access patterns, as an implementable approach, in one embodiment, for any given data item, the access pattern of the data item is determined based on its data characteristic information and data type, including:

[0103] Step 2021: For any data item, determine the access pattern analysis strategy for the data item based on its data type.

[0104] Step 2022: Analyze the access patterns of data items according to the data access pattern analysis strategy and the data characteristics of the data items.

[0105] Among them, data types are at least divided into two types: structured data and unstructured data.

[0106] It should be noted that structured data is typically stored in relational databases or similar formatted storage systems. This data has a clear schema and structure, making data cleaning relatively simple. Unstructured data, such as text, images, audio, and video, lacks a fixed structure, and its cleaning is relatively more difficult.

[0107] Specifically, in one embodiment, for any data item, if the data item is structured data, the access pattern analysis strategy for the data item can be determined based on the detailed type of the data item represented by the data feature information; if the data item is unstructured data, the access pattern analysis strategy for the data item can be determined based on the data format represented by the data feature information.

[0108] Among them, the detailed data item types are at least divided into two types: order data items and user information data items, and the data formats include image formats, video formats, and audio formats.

[0109] Specifically, access pattern analysis strategies for data items can be determined based on their detailed types or formats, so that the obtained access pattern analysis results match the actual types of the data items, thus improving the accuracy of the access pattern analysis results.

[0110] Specifically, for structured data, the storage system stores a large amount of structured business data, including order information (order data items), user information (user information data items), etc. This embodiment of the application analyzes access patterns based on the detailed types of data items, automatically detecting and identifying order data that has not been accessed for a long time, and periodically cleaning it up to free up significant storage space. Simultaneously, for user information data, the system determines the access patterns of user information data items based on a comprehensive analysis of access frequency and data age, ensuring that important data is not mistakenly deleted.

[0111] Specifically, regarding unstructured data, the storage system stores a large amount of unstructured business data, such as numerous news reports, video and audio files. This application embodiment analyzes the topics of news reports using Natural Language Processing (NLP) technology to predict their future access frequency, obtaining access pattern analysis and prediction results. For reports with predicted low access frequency, the system automatically performs compression or migration operations to free up storage space. Simultaneously, feature extraction technology is used to classify video and audio files, and corresponding cleaning strategies are formulated based on different categories of data, as the access patterns corresponding to different categories of data are similar.

[0112] Furthermore, in one embodiment, for any data item, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, the access frequency of the data item in a preset future period is predicted to obtain the future access frequency prediction result of the data item; based on the future access frequency prediction result of the data item, the access pattern of the data item is determined.

[0113] It should be noted that different access pattern analysis strategies for data items represent the differences in the prediction logic of the access frequency of data items within a preset future period. By using access pattern analysis strategies corresponding to different detailed data item types or data formats to predict access frequency, the reliability of future access frequency prediction results is improved, which in turn improves the accuracy of data item access pattern prediction results.

[0114] Specifically, access patterns can be determined based on current access frequency and data age. By monitoring and recording the current access frequency of data tables and fields (data items), data that has not been accessed for a long time can be identified and marked as potentially useless data. The age of the data can be calculated based on its creation and modification times, and data that has not been updated for a long time can be cleaned up. Combining these two methods can accurately predict the access patterns of the data items, effectively identifying and deleting data that occupies space but is no longer valuable. Regularly maintaining access pattern analysis strategies and other tasks ensures that data packets in the storage system remain within a reasonable range.

[0115] Specifically, for unstructured data, particularly text data, natural language processing (NLP) techniques can be used to analyze text content, extract key features, and predict data access patterns. For example, analyzing the topic of a text can determine its future reference value. Based on historical access records and machine learning models, the future access frequency of unstructured data can be predicted. Data with low predicted access frequency can be identified as potentially useless data. Feature extraction techniques can be used to classify unstructured data, such as content recognition for images and audio feature extraction for audio. Corresponding cleaning strategies can be developed based on different data categories. Through these methods, useless unstructured data can be effectively identified and deleted while ensuring data integrity, thus improving storage space utilization. For unstructured documents and emails, NLP and machine learning techniques can be used for cleaning.

[0116] Based on the above embodiments, to further improve the reliability of the target data cleaning strategy adopted in the embodiments of this application, and to further improve the data cleaning effect of the storage system, as an implementable approach, in one embodiment, the target data cleaning strategy of the storage system is determined according to the current business mode of the storage system and the access mode of each data item, including:

[0117] Step 2031: Based on the access patterns of each data item, filter out low-frequency access data items in the storage system;

[0118] Step 2032: Analyze the access cycle information of each low-frequency access data item to determine the last access time of each low-frequency access data item;

[0119] Step 2033: Based on the last access time of each low-frequency access data item, divide the low-frequency access data items into short-term low-frequency access data items and long-term low-frequency access data items.

[0120] Step 2034: When the current business mode of the storage system is normal mode, short-term low-frequency access data items are designated as data items to be deleted, and long-term low-frequency access data items are designated as data items to be migrated.

[0121] Step 2035: Determine the target data cleanup strategy for the storage system as deleting the data items to be deleted and migrating the data items to be migrated to the storage media.

[0122] Step 2036: When the current business mode of the storage system is high availability mode, short-term low-frequency access data items are designated as data items to be migrated, and long-term low-frequency access data items are designated as data items to be compressed.

[0123] Step 2037: Determine the target data cleaning strategy for the storage system as performing storage media migration processing on the data items to be migrated and data compression processing on the data items to be compressed.

[0124] Among these, the last access time of short-term low-frequency access data items is later than the last access time of long-term low-frequency access data items. Short-term low-frequency access data items are those that may still be accessed in the short term but with low access frequency, while long-term low-frequency access data items are those that have not been accessed for a long time and are predicted to have low future access frequency. Migrating the data items to be migrated to a storage medium can actually involve migrating the data items to a low-speed backend storage medium, such as an HDD or tape library.

[0125] In the storage media migration process for data items to be migrated, the backend storage media can be divided into multiple different tiers. Based on the current access frequency and importance of the data items, data can be stored on storage media with different performance and cost profiles. For example, non-critical or expired data from long-term infrequently accessed data items with relatively high current access frequency can be migrated to lower-cost storage media, while critical data from long-term infrequently accessed data items with relatively low current access frequency can be retained on high-performance storage media to ensure fast access.

[0126] It should be noted that in high-availability mode, storage systems need to be able to quickly restore service and reduce downtime in the event of a failure. Using short-term, low-frequency access data items as data to be migrated ensures that this data can be migrated between storage media, improving data availability and fault tolerance. High-availability mode emphasizes real-time data backup and protection to prevent data loss. Using long-term, low-frequency access data items as data to be compressed reduces data storage space usage and also improves data protection levels through compression. High-availability mode may sacrifice some performance for higher availability. Using short-term, low-frequency access data items as data to be migrated allows this data to be migrated to a more suitable storage medium, improving data access performance. High-availability mode requires more hardware, software, and maintenance costs; using long-term, low-frequency access data items as data to be compressed reduces data storage space usage, thereby improving the resource utilization of the storage system. The embodiments in this application consider the fault tolerance, data protection, performance optimization, and resource utilization characteristics of the storage system in high-availability mode to ensure that the storage system can operate efficiently and stably in high-availability mode.

[0127] The access patterns of data items are categorized into three types: high-frequency access pattern, low-frequency access pattern, and long-term inaccessible pattern. For high-frequency access data, a retention strategy is adopted; for low-frequency access data, a compression or migration strategy is adopted; and for long-term inaccessible data, a deletion strategy is adopted.

[0128] Specifically, in one embodiment, before deleting a data item, a snapshot of the data item to be deleted can be generated; in the event of accidental deletion of a data item, the data item can be recovered based on the snapshot data.

[0129] Specifically, to avoid accidental deletion of data items, this part of the data can be manually verified a second time before garbage collection (deleting data items). If the data is confirmed to be useful after the second verification, the deleted data items can be recovered through snapshot data.

[0130] Specifically, in one embodiment, during the data cleanup process of the storage system, a data cleanup operation log can be generated to facilitate auditing and backtracking. That is, after each cleanup operation is completed, an operation log is recorded, detailing information such as the identifier of the cleaned data, the type of cleanup operation, and the time. This ensures the effectiveness and adaptability of the cleanup strategy.

[0131] Based on the above embodiments, in order to improve the data security of the storage system, as an implementable approach, in one embodiment, the method further includes:

[0132] Step 301: Obtain the actual operating status of the storage system and the storage space usage.

[0133] Step 302: Determine the data cleaning and optimization strategy based on the actual operation of the storage system and the usage of storage space;

[0134] Step 303: Optimize the target data cleaning strategy according to the data cleaning and optimization strategy.

[0135] Specifically, the actual operating status of the storage system and storage space usage can be obtained through monitoring tools, system logs, and storage management software. Based on the data growth rate indicated by storage space usage and the actual operating status of the storage system, the frequency of data cleanup can be adjusted to ensure the rational utilization of storage space. The target data cleanup strategy includes the data cleanup frequency. Cleanup operations can also be suspended when a sudden increase in the access frequency of certain data items to be cleaned is detected. Alternatively, cleanup efforts can be increased when the storage system's storage space is nearing saturation, and cleanup conditions can be appropriately relaxed when storage space is ample.

[0136] Specifically, in one embodiment, the target data cleanup strategy can be optimized based on user feedback and changes in demand from the storage system. For example, protection mechanisms can be set up to prevent certain important data from being cleaned up.

[0137] The data cleaning method provided in this application involves acquiring data characteristic information, data type, and current business mode of each data item in the storage system; determining the access mode of any data item based on its data characteristic information and data type; determining the target data cleaning strategy for the storage system based on the current business mode and access modes of each data item; and cleaning the storage system according to the target data cleaning strategy. The method provided above improves data cleaning efficiency and reliability by predictively analyzing the access modes of data items and combining this with the current business mode of the storage system to determine the target data cleaning strategy. Furthermore, it boasts a high degree of automation, automatically detecting, identifying, and deleting useless data, reducing manual intervention and improving cleaning efficiency. It is highly adaptable, providing multiple cleaning strategies for different data types and access modes, and can dynamically adjust them to ensure their applicability. By combining NLP and machine learning technologies, it intelligently analyzes and processes data, accurately identifying useless data and reducing the risk of accidental deletion. Finally, it achieves high storage space utilization, effectively cleaning useless data and freeing up significant storage space, thus improving the utilization rate of the storage system. Low maintenance costs reduce the workload and cost of storage system maintenance, and improve the overall system performance and stability.

[0138] This application provides a data cleaning apparatus for performing the data cleaning method provided in the above embodiments.

[0139] like Figure 3 The diagram shown is a structural schematic of a data cleaning device provided in an embodiment of this application. The data cleaning device 30 includes: an acquisition module 301, a first determination module 302, a second determination module 303, and a cleaning module 304.

[0140] The system includes: an acquisition module for acquiring data characteristic information, data type, and current business mode of each data item in the storage system; a first determination module for determining the access mode of any data item based on its data characteristic information and data type; a second determination module for determining the target data cleaning strategy of the storage system based on its current business mode and the access mode of each data item; and a cleaning module for cleaning the storage system according to its target data cleaning strategy.

[0141] Specifically, in one embodiment, the first determining module is specifically used for:

[0142] For any given data item, determine the access pattern analysis strategy based on the data item's data type.

[0143] Based on the data item access pattern analysis strategy, analyze the data item access pattern according to the data item's data characteristic information;

[0144] Among them, data types are at least divided into two types: structured data and unstructured data.

[0145] Specifically, in one embodiment, the first determining module is specifically used for:

[0146] For any data item, if the data item is structured data, determine the access pattern analysis strategy for the data item based on the detailed type of the data item represented by the data feature information.

[0147] When the data item is unstructured, the access pattern analysis strategy for the data item is determined based on the data format represented by the data feature information of the data item.

[0148] Among them, the detailed data item types are at least divided into two types: order data items and user information data items, and the data formats include image formats, video formats, and audio formats.

[0149] Specifically, in one embodiment, the first determining module is specifically used for:

[0150] For any data item, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, the access frequency of the data item in the preset future period is predicted, and the future access frequency prediction result of the data item is obtained.

[0151] Based on the predicted future access frequency of data items, the access pattern of the data items is determined.

[0152] Specifically, in one embodiment, the second determining module is specifically used for:

[0153] Based on the access patterns of each data item, filter out low-frequency access data items in the storage system;

[0154] Analyze the access cycle information of each low-frequency access data item to determine the last access time of each low-frequency access data item;

[0155] Based on the last access time of each low-frequency access data item, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items.

[0156] When the current business mode of the storage system is normal mode, short-term low-frequency access data items are designated as data items to be deleted, and long-term low-frequency access data items are designated as data items to be migrated.

[0157] The target data cleanup strategy for the storage system is to delete the data items to be deleted and migrate the data items to be migrated to the storage media.

[0158] When the current business mode of the storage system is high availability mode, short-term low-frequency access data items are designated as data items to be migrated, and long-term low-frequency access data items are designated as data items to be compressed.

[0159] The target data cleanup strategy for the storage system is determined to be to perform storage media migration on the data items to be migrated and data compression on the data items to be compressed.

[0160] Among them, the last access time of short-term low-frequency access data items is later than the last access time of long-term low-frequency access data items.

[0161] Specifically, in one embodiment, the second determining module is further configured to:

[0162] Before deleting the data item to be deleted, generate a snapshot of the data item to be deleted;

[0163] In the event of accidental deletion of data items, the data items can be recovered based on the snapshot data.

[0164] Specifically, in one embodiment, the device further includes:

[0165] The optimization module is used to obtain the actual operating status and storage space usage of the storage system; determine the data cleaning and optimization strategy based on the actual operating status and storage space usage; and optimize the target data cleaning strategy based on the data cleaning and optimization strategy.

[0166] For example, as another logical functional division, such as Figure 4 The diagram shows another data cleaning device provided in this application embodiment. The device includes an intelligent data cleaning management module (IDC management module), a parameter module, a data monitoring module, a data analysis module, a log module, a cleaning execution module, a cleaning decision module, and a backend disk array module. The IDC management module is located on a board and primarily uses programmable logic devices such as ARM. The IDC management module can dynamically manage the data monitoring module, data analysis module, cleaning decision module, and cleaning execution module in both normal and high-availability modes. The data monitoring module is responsible for monitoring information such as data access frequency, creation and modification time. The data analysis module uses NLP and machine learning techniques to analyze data, extract features, and predict access patterns. The cleaning decision module dynamically selects a suitable cleaning strategy based on the data analysis results. The cleaning execution module executes data cleaning operations according to the instructions of the cleaning decision module, including identification, compression, migration, and deletion. The log recording module records the cleaning operations for auditing and backtracking.

[0167] Regarding the data cleaning device in this embodiment, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0168] The data cleaning device provided in this application embodiment is used to execute the data cleaning method provided in the above embodiment. Its implementation method and principle are the same, and will not be described again.

[0169] This application provides an electronic device for performing the data cleaning method provided in the above embodiments.

[0170] like Figure 5 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. The electronic device 50 includes at least one processor 51 and a memory 52.

[0171] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the data cleanup method provided in the above embodiments.

[0172] The electronic device provided in this application embodiment is used to execute the data cleaning method provided in the above embodiment. Its implementation method and principle are the same, and will not be described again.

[0173] This application provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the data cleanup method provided in any of the above embodiments.

[0174] The storage medium containing computer-executable instructions provided in this application embodiment can be used to store computer-executable instructions for the data cleaning method provided in the foregoing embodiments. Its implementation method and principle are the same, and will not be described again.

[0175] This application provides a computer program product, including computer instructions, which are used to cause a computer to execute the data cleaning method provided in the foregoing embodiments.

[0176] The computer program product provided in this application embodiment can be used to execute the computer instructions of the data cleaning method provided in the foregoing embodiment. Its implementation method and principle are the same, and will not be described again.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0180] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the present invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0182] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data cleaning method, characterized in that, include: Obtain the data characteristic information of each data item in the storage system, the data type of each data item, and the current business mode of the storage system; For any of the data items, the access mode of the data item is determined based on the data feature information and data type of the data item; Based on the current business mode of the storage system and the access mode of each data item, determine the target data cleanup strategy for the storage system; Data cleaning is performed on the storage system according to the target data cleaning strategy of the storage system; The step of determining the target data cleanup strategy for the storage system based on the current business mode of the storage system and the access mode of each data item includes: Based on the access patterns of each data item, low-frequency access data items are filtered in the storage system; Analyze the access cycle information of each of the low-frequency access data items to determine the last access time of each of the low-frequency access data items; Based on the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items. When the current business mode of the storage system is normal mode, the short-term low-frequency access data items are designated as data items to be deleted, and the long-term low-frequency access data items are designated as data items to be migrated. The target data cleanup strategy for the storage system is determined to be to delete the data items to be deleted and to migrate the data items to be migrated to the storage medium. When the current business mode of the storage system is high availability mode, the short-term low-frequency access data items are regarded as data items to be migrated, and the long-term low-frequency access data items are regarded as data items to be compressed. The target data cleanup strategy for the storage system is determined to be to perform storage medium migration processing on the data items to be migrated and data compression processing on the data items to be compressed. The last access time of the short-term low-frequency access data item is later than the last access time of the long-term low-frequency access data item.

2. The method according to claim 1, characterized in that, For any of the data items, determining the access mode of the data item based on its data characteristics and data type includes: For any of the data items, determine the access pattern analysis strategy for the data item based on the data item's data type; According to the access pattern analysis strategy of the data item, the access pattern of the data item is analyzed based on the data feature information of the data item; The data types mentioned above are at least divided into two types: structured data and unstructured data.

3. The method according to claim 2, characterized in that, The step of determining the access pattern analysis strategy for any of the data items based on their data type includes: For any of the data items, if the data item is structured data, an access pattern analysis strategy for the data item is determined based on the detailed type of the data item characterized by the data feature information of the data item. In the case that the data item is unstructured data, an access pattern analysis strategy for the data item is determined based on the data format represented by the data feature information of the data item. The detailed types of the data items are at least divided into two types: order data items and user information data items. The data formats include image formats, video formats, and audio formats.

4. The method according to claim 3, characterized in that, The access pattern analysis strategy for the data item, which analyzes the access pattern of the data item based on its data characteristic information, includes: For any of the data items, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, the access frequency of the data item in a preset future period is predicted, and the future access frequency prediction result of the data item is obtained. The access pattern of the data item is determined based on the predicted future access frequency of the data item.

5. The method according to claim 1, characterized in that, The method further includes: Before deleting the data item to be deleted, a snapshot of the data item to be deleted is generated; In the event of accidental deletion of data items, the data items can be recovered based on the snapshot data.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the actual operating status and storage space usage of the storage system; Based on the actual operation of the storage system and the usage of storage space, determine the data cleaning and optimization strategy; Based on the data cleaning and optimization strategy, optimize the target data cleaning strategy.

7. A data cleaning device, characterized in that, include: The acquisition module is used to acquire data feature information of each data item in the storage system, the data type of each data item, and the current business mode of the storage system; The first determining module is used to determine the access mode of any data item based on its data feature information and data type. The second determining module is used to determine the target data cleaning strategy of the storage system based on the current business mode of the storage system and the access mode of each data item. The cleaning module is used to clean the data in the storage system according to the target data cleaning strategy of the storage system; The second determining module is specifically used for: Based on the access patterns of each data item, low-frequency access data items are filtered in the storage system; Analyze the access cycle information of each of the low-frequency access data items to determine the last access time of each of the low-frequency access data items; Based on the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items. When the current business mode of the storage system is normal mode, the short-term low-frequency access data items are designated as data items to be deleted, and the long-term low-frequency access data items are designated as data items to be migrated. The target data cleanup strategy for the storage system is determined to be to delete the data items to be deleted and to migrate the data items to be migrated to the storage medium. When the current business mode of the storage system is high availability mode, the short-term low-frequency access data items are regarded as data items to be migrated, and the long-term low-frequency access data items are regarded as data items to be compressed. The target data cleanup strategy for the storage system is determined to be to perform storage medium migration processing on the data items to be migrated and data compression processing on the data items to be compressed. The last access time of the short-term low-frequency access data item is later than the last access time of the long-term low-frequency access data item.

8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 6.

Images