Data cleaning method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a data cleaning method and device, electronic equipment and a storage medium, the method comprises the following steps: obtaining data feature information of each data item in a storage system, a data type of each data item and a current service mode of the storage system; for any data item, determining an access mode of the data item according to the data feature information and the data type of the data item; determining a target data cleaning strategy of the storage system according to the current service mode of the storage system and the access mode of each data item; and performing data cleaning on the storage system according to the target data cleaning strategy of the storage system. According to the method provided by the scheme, the access mode of the data item is predicted and analyzed, and the target data cleaning strategy is determined in combination with the current service mode of the storage system, so that the reliability of a data cleaning result is improved while the data cleaning efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data cleaning method, device, electronic device and storage medium. Background Art

[0002] At present, with the rapid development of technologies such as big data and cloud computing, storage systems, especially centralized storage systems, are under tremendous pressure. After long-term application, storage systems will accumulate a large amount of redundant data and useless data, causing a waste of storage resources of storage systems. Therefore, how to clean up data in storage systems has become a key research topic.

[0003] In the related art, manual operation is usually relied on, where technicians screen redundant data and useless data waiting for data cleaning in the storage system based on their own experience, and clean up the data to free up storage space of the storage system.

[0004] However, since the manual operation process is relatively cumbersome and it is difficult for technical personnel to accurately screen the data to be cleaned, data may be mistakenly cleaned or the data to be cleaned may be incompletely screened, which not only reduces the efficiency of data cleaning, but also cannot guarantee the reliability of data cleaning results. Summary of the invention

[0005] The present application provides a data cleaning method, device, electronic device and storage medium to solve the defects of related technologies, such as not only reducing the data cleaning efficiency but also failing to ensure the reliability of data cleaning results.

[0006] The first aspect of the present application provides a data cleaning method, comprising:

[0007] Acquire data feature information of each data item in the storage system, the data type of each data item and the current business mode of the storage system;

[0008] For any of the data items, determining an access mode of the data item according to data feature information and a data type of the data item;

[0009] Determining a target data cleaning strategy for the storage system according to a current business mode of the storage system and an access mode of each of the data items;

[0010] The storage system is cleaned of data according to a target data cleanup strategy of the storage system.

[0011] In an optional implementation manner, for any of the data items, determining the access mode of the data item according to the data feature information and the data type of the data item includes:

[0012] For any of the data items, determining an access pattern analysis strategy for the data item according to the data type of the data item;

[0013] Analyzing the access pattern of the data item according to the access pattern analysis strategy of the data item and based on the data feature information of the data item;

[0014] The data types are divided into at least two types: structured data and unstructured data.

[0015] In an optional implementation, for any of the data items, determining the access pattern analysis strategy of the data item according to the data type of the data item includes:

[0016] For any of the data items, when the data item is structured data, determining an access pattern analysis strategy for the data item according to a detailed type of the data item represented by data feature information of the data item;

[0017] In the case where the data item is unstructured data, determining an access pattern analysis strategy for the data item according to a data format represented by data feature information of the data item;

[0018] The detailed types of the data items are divided into at least two types: order data items and user information data items, and the data formats include image formats, video formats and audio formats.

[0019] In an optional implementation manner, analyzing the access pattern of the data item according to the access pattern analysis strategy of the data item and based on the data feature information of the data item includes:

[0020] For any of the data items, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, predict the access frequency of the data item in a preset future period, and obtain the future access frequency prediction result of the data item;

[0021] An access pattern of the data item is determined according to a prediction result of the future access frequency of the data item.

[0022] In an optional implementation, determining a target data cleaning strategy for the storage system according to a current business mode of the storage system and an access mode of each of the data items includes:

[0023] screening low-frequency access data items in the storage system according to the access pattern of each of the data items;

[0024] Analyzing the access cycle information of each of the infrequently accessed data items to determine the last access time of each of the infrequently accessed data items;

[0025] According to the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items;

[0026] In the case where the current business mode of the storage system is the normal mode, the short-term low-frequency access data items are used as data items to be deleted, and the long-term low-frequency access data items are used as data items to be migrated;

[0027] Determining a target data cleaning strategy of the storage system is to delete the data items to be deleted and to migrate the data items to be migrated;

[0028] In a case where the current business mode of the storage system is a high availability mode, the short-term low-frequency access data items are used as data items to be migrated, and the long-term low-frequency access data items are used as data items to be compressed;

[0029] Determining a target data cleaning strategy of the storage system is to perform storage medium migration processing on the data items to be migrated and to perform data compression processing on the data items to be compressed;

[0030] The last access time of the short-term infrequently accessed data item is later than the last access time of the long-term infrequently accessed data item.

[0031] In an optional embodiment, the method further includes:

[0032] Before deleting the data item to be deleted, generating snapshot data of the data item to be deleted;

[0033] In the event that a data item is accidentally deleted, the data item is restored based on the snapshot data.

[0034] In an optional embodiment, the method further includes:

[0035] Obtaining actual operation status and storage space usage of the storage system;

[0036] Determine a data cleaning optimization strategy based on the actual operation of the storage system and the storage space usage;

[0037] According to the data cleaning optimization strategy, the target data cleaning strategy is optimized.

[0038] A second aspect of the present application provides a data cleaning device, comprising:

[0039] An acquisition module, used to acquire data feature information of each data item in the storage system, a data type of each data item and a current business mode of the storage system;

[0040] A first determination module, configured to determine, for any of the data items, an access mode of the data item according to data feature information and a data type of the data item;

[0041] A second determination module, configured to determine a target data cleaning strategy for the storage system according to a current business mode of the storage system and an access mode of each of the data items;

[0042] The cleaning module is used to clean up data in the storage system according to the target data cleaning strategy of the storage system.

[0043] In an optional implementation manner, the first determining module is specifically configured to:

[0044] For any of the data items, determining an access pattern analysis strategy for the data item according to the data type of the data item;

[0045] Analyzing the access pattern of the data item according to the access pattern analysis strategy of the data item and based on the data feature information of the data item;

[0046] The data types are divided into at least two types: structured data and unstructured data.

[0047] In an optional implementation manner, the first determining module is specifically configured to:

[0048] For any of the data items, when the data item is structured data, determining an access pattern analysis strategy for the data item according to a detailed type of the data item represented by data feature information of the data item;

[0049] In the case where the data item is unstructured data, determining an access pattern analysis strategy for the data item according to a data format represented by data feature information of the data item;

[0050] The detailed types of the data items are divided into at least two types: order data items and user information data items, and the data formats include image formats, video formats and audio formats.

[0051] In an optional implementation manner, the first determining module is specifically configured to:

[0052] For any of the data items, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, predict the access frequency of the data item in a preset future period, and obtain the future access frequency prediction result of the data item;

[0053] An access pattern of the data item is determined according to a prediction result of the future access frequency of the data item.

[0054] In an optional implementation manner, the second determining module is specifically configured to:

[0055] screening low-frequency access data items in the storage system according to the access pattern of each of the data items;

[0056] Analyzing the access cycle information of each of the infrequently accessed data items to determine the last access time of each of the infrequently accessed data items;

[0057] According to the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items;

[0058] In the case where the current business mode of the storage system is the normal mode, the short-term low-frequency access data items are used as data items to be deleted, and the long-term low-frequency access data items are used as data items to be migrated;

[0059] Determining a target data cleaning strategy of the storage system is to delete the data items to be deleted and to migrate the data items to be migrated;

[0060] In a case where the current business mode of the storage system is a high availability mode, the short-term low-frequency access data items are used as data items to be migrated, and the long-term low-frequency access data items are used as data items to be compressed;

[0061] Determining a target data cleaning strategy of the storage system is to perform storage medium migration processing on the data items to be migrated and to perform data compression processing on the data items to be compressed;

[0062] The last access time of the short-term infrequently accessed data item is later than the last access time of the long-term infrequently accessed data item.

[0063] In an optional implementation manner, the second determining module is further configured to:

[0064] Before deleting the data item to be deleted, generating snapshot data of the data item to be deleted;

[0065] In the event that a data item is accidentally deleted, the data item is restored based on the snapshot data.

[0066] In an optional embodiment, the device further comprises:

[0067] The optimization module is used to obtain the actual operation status and storage space usage of the storage system; determine the data cleaning optimization strategy according to the actual operation status and storage space usage of the storage system; and optimize the target data cleaning strategy according to the data cleaning optimization strategy.

[0068] A third aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0069] The memory stores computer-executable instructions;

[0070] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0071] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0072] A fifth aspect of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method described in the first aspect and various possible designs of the first aspect.

[0073] The technical solution of this application has the following advantages:

[0074] The present application provides a data cleaning method, device, electronic device and storage medium, the method comprising: obtaining data feature information of each data item in a storage system, the data type of each data item and the current business mode of the storage system; for any data item, determining the access mode of the data item according to the data feature information and data type of the data item; determining the target data cleaning strategy of the storage system according to the current business mode of the storage system and the access mode of each data item; and performing data cleaning on the storage system according to the target data cleaning strategy of the storage system. The method provided by the above scheme improves the reliability of the data cleaning result while improving the data cleaning efficiency by predicting and analyzing the access mode of the data item and determining the target data cleaning strategy in combination with the current business mode of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0076] Figure 1 A schematic diagram of the structure of a data cleaning system based on an embodiment of the present application;

[0077] Figure 2 A flowchart of a data cleaning method provided in an embodiment of the present application;

[0078] Figure 3 A schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application;

[0079] Figure 4 A schematic diagram of the structure of another data cleaning device provided in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0081] The above drawings have shown clear embodiments of the present application, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0084] In related technologies, the rapid development of information technology is inevitably accompanied by the increasing demand for the generation and storage of various types of data, especially the development of big data and cloud computing, which brings increasing pressure to storage systems, especially high-end centralized storage systems. Whether it is structured data or unstructured data, a large amount of redundant and useless data will accumulate over time. These useless data not only occupy valuable storage space, but also increase the maintenance cost of the system and reduce the overall storage efficiency. Traditional data cleaning methods usually rely on manual operations or simple automated scripts, which have problems such as low efficiency, easy errors, and inability to quickly adapt to different data types and access modes, and cannot restore data in time after errors. Therefore, there is an urgent need for an efficient, intelligent and adaptable data cleaning mechanism to improve the utilization of storage systems and reduce maintenance costs. At present, there is no independent module in the industry to uniformly manage this intelligent data cleaning mechanism, which can easily lead to inefficient cluster response to front-end services in some extreme scenarios, thereby affecting the availability of the entire system.

[0085] In response to the above problems, embodiments of the present application provide a data cleaning method, device, electronic device, and storage medium, the method comprising: obtaining data feature information of each data item in a storage system, the data type of each data item, and the current business mode of the storage system; for any data item, determining the access mode of the data item based on the data feature information and the data type of the data item; determining the target data cleaning strategy of the storage system based on the current business mode of the storage system and the access mode of each data item; and performing data cleaning on the storage system according to the target data cleaning strategy of the storage system. The method provided by the above scheme improves the reliability of the data cleaning results while improving the efficiency of data cleaning by predicting and analyzing the access mode of the data item and determining the target data cleaning strategy in combination with the current business mode of the storage system.

[0086] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0087] First, the structure of the data cleaning system on which this application is based is described:

[0088] The data cleaning method, device, electronic device and storage medium provided in the embodiments of the present application are suitable for screening data items to be cleaned in a storage system and performing corresponding cleaning processing on the data items to be cleaned. Figure 1As shown, it is a structural diagram of the data cleaning system based on the embodiment of the present application, which mainly includes a data acquisition device, a data storage system and a data cleaning device. Among them, the data acquisition device is used to collect data feature information of each data item in the storage system, the data type of each data item and the current business mode of the storage system, and send the collected information to the data cleaning device. The data cleaning state formulates the target data cleaning strategy of the storage system according to the obtained information, and then cleans the storage system according to the target data cleaning strategy to release the storage space of the storage system.

[0089] The embodiment of the present application provides a data cleaning method for screening data items to be cleaned in a storage system and performing corresponding cleaning processing on the data items to be cleaned. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used for data cleaning.

[0090] like Figure 2 FIG. 1 is a flow chart of a data cleaning method provided in an embodiment of the present application, the method comprising:

[0091] Step 201: Acquire data feature information of each data item in the storage system, the data type of each data item, and the current business mode of the storage system.

[0092] Specifically, the data items of the storage system can be monitored in real time to record the data feature information of each data item in the storage system in real time, and the data type of the data item and the current business mode of the storage system can be collected and recorded.

[0093] Step 202: for any data item, determine the access mode of the data item according to the data feature information and data type of the data item.

[0094] The data feature information of the data item includes current access frequency, creation time, modification time, etc. The data type is divided into at least two types: structured data and unstructured data.

[0095] Specifically, by analyzing the data feature information and data type of the data item, access pattern feature information such as the future access frequency of the data item can be predicted to obtain the access pattern of the data item.

[0096] Step 203: Determine a target data cleaning strategy for the storage system according to the current business mode of the storage system and the access mode of each data item.

[0097] Specifically, the data items to be cleaned can be located in the storage system according to the current business mode of the storage system and the access mode of each data item, and then a specific target data cleaning strategy can be determined according to the data items to be cleaned.

[0098] The target data cleaning strategy includes a data cleaning method for the data items to be cleaned.

[0099] Step 204: clean up data on the storage system according to the target data cleanup strategy of the storage system.

[0100] Specifically, according to the data cleaning method for each data item to be cleaned represented by the target data cleaning strategy, corresponding data cleaning operations can be performed on the data item to be cleaned, so as to achieve data cleaning of the storage system and release storage space of the storage system.

[0101] Specifically, in one embodiment, in order to further release storage space of the storage system, duplicate data copies in the storage system may also be cleaned up to remove duplicate data in the storage system, reduce waste of storage space, and improve space utilization.

[0102] On the basis of the above embodiments, since a large number of data items are stored in the storage system and the data items have a certain diversity, in order to ensure the accuracy of the access pattern analysis and prediction results of the data items, as an implementable method, in one embodiment, for any data item, according to the data feature information and data type of the data item, the access pattern of the data item is determined, including:

[0103] Step 2021, for any data item, determine the access pattern analysis strategy of the data item according to the data type of the data item;

[0104] Step 2022, according to the access pattern analysis strategy of the data item, analyze the access pattern of the data item according to the data feature information of the data item.

[0105] Among them, data types are divided into at least two types: structured data and unstructured data.

[0106] It should be noted that structured data is usually stored in relational databases or similar formatted storage systems. Its data has a clear pattern and structure, and data cleaning is relatively simple. Unstructured data is text, images, audio, video and other data that has no fixed structure, and it is relatively difficult to clean up unstructured data.

[0107] Specifically, in one embodiment, for any data item, when the data item is structured data, the access pattern analysis strategy of the data item can be determined based on the detailed type of the data item represented by the data feature information of the data item; when the data item is unstructured data, the access pattern analysis strategy of the data item can be determined based on the data format represented by the data feature information of the data item.

[0108] The detailed types of data items are divided into at least two types: order data items and user information data items, and the data formats include image formats, video formats and audio formats.

[0109] Specifically, the access pattern analysis strategy of the data item can be determined according to the detailed type or data format of the data item, so that the obtained access pattern analysis result matches the actual type of the data item, and the accuracy of the access pattern analysis result is improved.

[0110] Specifically, for structured data, the storage system stores a large amount of structured business data, including order information (order data items), user information (user information data items), etc. The embodiment of the present application can automatically detect and identify order data that has not been accessed for a long time by performing access pattern analysis based on the detailed type of the data item, and regularly clean it up to free up a large amount of storage space. At the same time, for user information data, the system determines the access pattern of the user information data item based on the comprehensive analysis results of the access frequency and data age to ensure that important data will not be deleted by mistake.

[0111] Specifically, for unstructured data, the storage system stores a large amount of unstructured business data, such as a large number of news reports, video and audio files. The embodiment of the present application analyzes the subject of news reports through natural language processing (NLP) technology, predicts its future access frequency, and obtains access pattern analysis prediction results. For reports with low predicted access frequency, the system will automatically perform compression or migration operations to release storage space. At the same time, feature extraction technology is used to classify video and audio files, and corresponding cleaning strategies are formulated according to different categories of data. The access patterns corresponding to different categories of data are similar.

[0112] Furthermore, in one embodiment, for any data item, according to the access pattern analysis strategy of the data item, the access frequency of the data item in a preset future period is predicted based on the current access frequency, creation time and modification time represented by the data feature information of the data item, and the future access frequency prediction result of the data item is obtained; based on the future access frequency prediction result of the data item, the access pattern of the data item is determined.

[0113] It should be noted that different access pattern analysis strategies for data items represent the differences in the prediction logic of the access frequency of data items in a preset future period. By adopting access pattern analysis strategies corresponding to different data item detailed types or data formats to predict the access frequency, the reliability of the future access frequency prediction results is improved, thereby improving the accuracy of the access pattern prediction results of the data items.

[0114] Specifically, the access pattern can be determined based on the current access frequency and data age. By monitoring and recording the current access frequency of data tables and fields (data items), data that has not been accessed for a long time can be identified and marked as potentially useless data. The age of the data is calculated based on the creation and modification time of the data, and data that has not been updated for a long time can be cleaned up. Combining these two methods can accurately predict the access pattern of the data item to effectively identify and delete data that takes up space but is no longer valuable. Through regular maintenance of access pattern analysis strategies and other tasks, ensure that the data packets in the storage system always remain within a reasonable range.

[0115] Specifically, for unstructured data and text data, natural language processing technology can also be used to analyze text content, extract key features, and predict data access patterns. For example, by analyzing the subject of the text, it can be determined whether it will still have reference value in the future. Based on historical access records and machine learning models, the future access frequency of unstructured data can be predicted. For data with a low predicted access frequency, it can be identified as potentially useless data. Unstructured data can be classified using feature extraction technology, such as content recognition of images and audio feature extraction of audio. Corresponding cleanup strategies are formulated according to different categories of data. Through these methods, useless unstructured data can be effectively identified and deleted while ensuring data integrity, thereby improving storage space utilization. For unstructured documents and emails, NLP and machine learning technologies are used for cleanup.

[0116] On the basis of the above embodiments, in order to further improve the reliability of the target data cleaning strategy adopted in the embodiments of the present application, so as to further improve the data cleaning effect of the storage system, as an implementable manner, in one embodiment, according to the current business mode of the storage system and the access mode of each data item, the target data cleaning strategy of the storage system is determined, including:

[0117] Step 2031, screening low-frequency access data items in the storage system according to the access mode of each data item;

[0118] Step 2032, analyzing the access cycle information of each low-frequency access data item to determine the last access time of each low-frequency access data item;

[0119] Step 2033, dividing the low-frequency access data items into short-term low-frequency access data items and long-term low-frequency access data items according to the last access time of each low-frequency access data item;

[0120] Step 2034: when the current service mode of the storage system is the normal mode, the short-term low-frequency access data items are taken as data items to be deleted, and the long-term low-frequency access data items are taken as data items to be migrated;

[0121] Step 2035, determining that the target data cleaning strategy of the storage system is to perform a deletion process on the data items to be deleted and to perform a storage medium migration process on the data items to be migrated;

[0122] Step 2036: when the current service mode of the storage system is the high availability mode, the short-term low-frequency access data items are used as the data items to be migrated, and the long-term low-frequency access data items are used as the data items to be compressed;

[0123] Step 2037, determining that the target data cleaning strategy of the storage system is to perform storage medium migration processing on the data items to be migrated and to perform data compression processing on the data items to be compressed.

[0124] Among them, the last access time of the short-term low-frequency access data item is later than the last access time of the long-term low-frequency access data item. The short-term low-frequency access data item is data that may still be accessed in the short term but the access frequency is not high, and the long-term low-frequency access data item is data that has not been accessed for a long time and is predicted to have a low access frequency in the future. The storage medium migration processing of the data item to be migrated can actually be to migrate the data item to be migrated to a low-speed backend storage medium, such as an HDD or a tape library.

[0125] Among them, for the storage medium migration processing of the data items to be migrated, the backend storage medium can also be divided into multiple different levels, and the data can be stored on storage media with different performance and costs according to the current access frequency and importance of the data items to be migrated. For example, non-critical data or expired data of long-term low-frequency access data items with relatively high current access frequency can be migrated to storage media with lower costs, and critical data of long-term low-frequency access data items with relatively low current access frequency can be retained on high-performance storage media to ensure fast access.

[0126] It should be noted that the storage system needs to be able to quickly restore services and reduce downtime when a failure occurs in the high-availability mode. Taking short-term low-frequency access data items as data items to be migrated can ensure that these data are migrated between storage media to improve the availability and fault tolerance of data. The high-availability mode emphasizes real-time backup and protection of data to prevent data loss. Taking long-term low-frequency access data items as data items to be compressed can reduce the occupation of data storage space, and at the same time, the protection level of data can be improved by data compression. The high-availability mode may sacrifice some performance to a certain extent in exchange for higher availability. Taking short-term low-frequency access data items as data items to be migrated can migrate these data to more suitable storage media to improve data access performance. The high-availability mode requires more hardware, software and maintenance costs. Taking long-term low-frequency access data items as data items to be compressed can reduce the occupation of data storage space, thereby improving the resource utilization of the storage system. The embodiment of the present application takes into account the fault tolerance, data protection, performance optimization and resource utilization of the storage system in the high-availability mode to ensure that the storage system can run efficiently and stably in the high-availability mode.

[0127] Among them, the access patterns of data items are divided into three types: high-frequency access pattern, low-frequency access pattern and long-term non-access pattern. For high-frequency access data, a retention strategy is adopted; for low-frequency access data, a compression or migration strategy is adopted; for long-term non-access data, a deletion strategy is adopted.

[0128] Specifically, in one embodiment, before deleting a data item to be deleted, snapshot data of the data item to be deleted may be generated; in the event that a data item is deleted by mistake, the data item is restored based on the snapshot data.

[0129] Specifically, to avoid accidental deletion of data items, this part of data can also be manually re-confirmed before garbage collection (deleting data items). If the deleted data is determined to be useful data after secondary confirmation, the deleted data items can be restored through snapshot data.

[0130] Specifically, in one embodiment, during the data cleaning process of the storage system, a data cleaning operation log may be generated for auditing and backtracking. That is, after each cleaning operation is completed, an operation log is recorded, which records in detail information such as the identification of the cleaned data, the type and time of the cleaning operation, etc., in order to ensure the effectiveness and adaptability of the cleaning strategy.

[0131] On the basis of the above embodiment, in order to improve the data security of the storage system, as an implementable manner, in one embodiment, the method further includes:

[0132] Step 301, obtaining the actual operation status of the storage system and the storage space usage;

[0133] Step 302, determining a data cleaning optimization strategy according to the actual operation of the storage system and the storage space usage;

[0134] Step 303: Optimize the target data cleaning strategy according to the data cleaning optimization strategy.

[0135] Specifically, the actual operation of the storage system and the usage of storage space can be obtained through monitoring tools, system logs, and storage management software. Specifically, the frequency of data cleaning can be adjusted according to the usage of storage space and the data growth rate represented by the actual operation of the storage system to ensure the rational use of storage space. Among them, the target data cleaning strategy includes the frequency of data cleaning. It is also possible to suspend the cleaning operation when it is detected that the access frequency of certain data items to be cleaned suddenly increases. It is also possible to increase the cleaning intensity when the storage space of the storage system is close to saturation; when the storage space is sufficient, the cleaning conditions can be appropriately relaxed.

[0136] Specifically, in one embodiment, the target data cleaning strategy can also be optimized according to the usage feedback and demand changes of the storage system users. For example, a protection mechanism can be set for some important data to prevent it from being cleaned up.

[0137] The data cleaning method provided in the embodiment of the present application obtains the data feature information of each data item in the storage system, the data type of each data item and the current business mode of the storage system; for any data item, according to the data feature information and data type of the data item, determines the access mode of the data item; according to the current business mode of the storage system and the access mode of each data item, determines the target data cleaning strategy of the storage system; and performs data cleaning on the storage system according to the target data cleaning strategy of the storage system. The method provided by the above scheme predicts and analyzes the access mode of the data item, and determines the target data cleaning strategy in combination with the current business mode of the storage system, thereby improving the efficiency of data cleaning and the reliability of the data cleaning result. In addition, the degree of automation is high, and manual intervention is reduced and the cleaning efficiency is improved by automatically detecting, identifying and deleting useless data. It has strong adaptability, and provides a variety of cleaning strategies for different data types and access modes, and can be adjusted dynamically to ensure the applicability of the cleaning strategy. Combined with NLP and machine learning technology, the data is intelligently analyzed and processed, and useless data is accurately identified to reduce the risk of accidental deletion. The storage space utilization rate is high, and a large amount of storage space is released by effectively cleaning useless data, thereby improving the utilization rate of the storage system. Low maintenance cost reduces the maintenance workload and cost of the storage system and improves the performance and stability of the overall system.

[0138] An embodiment of the present application provides a data cleaning device for executing the data cleaning method provided in the above embodiment.

[0139] like Figure 3 , which is a schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application. The data cleaning device 30 comprises: an acquisition module 301 , a first determination module 302 , a second determination module 303 and a cleaning module 304 .

[0140] Among them, the acquisition module is used to obtain the data feature information of each data item in the storage system, the data type of each data item and the current business mode of the storage system; the first determination module is used to determine the access mode of the data item according to the data feature information and data type of the data item for any data item; the second determination module is used to determine the target data cleaning strategy of the storage system according to the current business mode of the storage system and the access mode of each data item; the cleaning module is used to clean the data of the storage system according to the target data cleaning strategy of the storage system.

[0141] Specifically, in one embodiment, the first determining module is specifically configured to:

[0142] For any data item, determine the access pattern analysis strategy of the data item according to the data type of the data item;

[0143] According to the access pattern analysis strategy of the data item, the access pattern of the data item is analyzed according to the data feature information of the data item;

[0144] Among them, data types are divided into at least two types: structured data and unstructured data.

[0145] Specifically, in one embodiment, the first determining module is specifically configured to:

[0146] For any data item, when the data item is structured data, an access pattern analysis strategy for the data item is determined according to a detailed type of the data item represented by data feature information of the data item;

[0147] In the case where the data item is unstructured data, determining an access pattern analysis strategy for the data item according to a data format represented by data feature information of the data item;

[0148] The detailed types of data items are divided into at least two types: order data items and user information data items, and the data formats include image formats, video formats and audio formats.

[0149] Specifically, in one embodiment, the first determining module is specifically configured to:

[0150] For any data item, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, the access frequency of the data item in a preset future period is predicted to obtain the future access frequency prediction result of the data item;

[0151] The access pattern of the data item is determined based on the prediction result of the future access frequency of the data item.

[0152] Specifically, in one embodiment, the second determining module is specifically configured to:

[0153] Filter low-frequency access data items in the storage system based on the access patterns of each data item;

[0154] Analyze the access cycle information of each low-frequency access data item to determine the last access time of each low-frequency access data item;

[0155] According to the last access time of each low-frequency access data item, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items;

[0156] When the current business mode of the storage system is the normal mode, the short-term low-frequency access data items are taken as the data items to be deleted, and the long-term low-frequency access data items are taken as the data items to be migrated;

[0157] Determine the target data cleanup strategy of the storage system to delete the data items to be deleted and migrate the storage media of the data items to be migrated;

[0158] When the current business mode of the storage system is a high availability mode, short-term low-frequency access data items are used as data items to be migrated, and long-term low-frequency access data items are used as data items to be compressed;

[0159] Determine the target data cleaning strategy of the storage system as performing storage medium migration processing on the data items to be migrated and performing data compression processing on the data items to be compressed;

[0160] The last access time of the short-term low-frequency access data item is later than the last access time of the long-term low-frequency access data item.

[0161] Specifically, in one embodiment, the second determining module is further used to:

[0162] Before deleting the data item to be deleted, generating snapshot data of the data item to be deleted;

[0163] In the event that a data item is accidentally deleted, the data item is restored based on the snapshot data.

[0164] Specifically, in one embodiment, the device further includes:

[0165] The optimization module is used to obtain the actual operation status of the storage system and the storage space usage; determine the data cleaning optimization strategy based on the actual operation status of the storage system and the storage space usage; and optimize the target data cleaning strategy based on the data cleaning optimization strategy.

[0166] For example, as another division of logical functions, Figure 4 As shown, it is a structural schematic diagram of another data cleaning device provided by an embodiment of the present application, and the device includes an intelligent data cleaning management module (IDC management module), a parameter module, a data monitoring module, a data analysis module, a log module, a cleaning execution module, a cleaning decision module and a back-end disk array module. The IDC management module is located on the board, and mainly applies programmable logic devices such as ARM. The IDC management module can dynamically manage the data monitoring module, the data analysis module, the cleaning decision module and the cleaning execution module in normal mode and high availability mode respectively. Among them, the data monitoring module is responsible for monitoring the access frequency, creation and modification time of the data and other information. The data analysis module uses NLP and machine learning technology to analyze the data, extract features, and predict access patterns. The cleaning decision module dynamically selects a suitable cleaning strategy based on the data analysis results. Cleaning execution module: according to the instructions of the cleaning decision module, perform data cleaning operations, including operations such as identification, compression, migration and deletion. The logging module records the logs of the cleaning operation for auditing and backtracking.

[0167] Regarding the data cleaning device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0168] The data cleaning device provided in the embodiment of the present application is used to execute the data cleaning method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0169] An embodiment of the present application provides an electronic device for executing the data cleaning method provided in the above embodiment.

[0170] like Figure 5 FIG. 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 50 includes: at least one processor 51 and a memory 52.

[0171] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data cleaning method provided in the above embodiment.

[0172] The electronic device provided in the embodiment of the present application is used to execute the data cleaning method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0173] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, a data cleaning method provided in any of the above embodiments is implemented.

[0174] The storage medium containing computer executable instructions provided in the embodiments of the present application can be used to store computer executable instructions of the data cleaning method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0175] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the data cleaning method provided in the aforementioned embodiment.

[0176] The embodiments of the present application provide a computer program product that can be used to execute computer instructions of the data cleaning method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0180] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0181] A part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0182] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data cleaning method, characterized in that: include: Acquire data feature information of each data item in the storage system, the data type of each data item and the current business mode of the storage system; For any of the data items, determining an access mode of the data item according to data feature information and a data type of the data item; Determining a target data cleaning strategy for the storage system according to a current business mode of the storage system and an access mode of each of the data items; The storage system is cleaned of data according to a target data cleanup strategy of the storage system.

2. The method according to claim 1, characterized in that The step of determining, for any of the data items, an access mode of the data item according to data feature information and a data type of the data item, includes: For any of the data items, determining an access pattern analysis strategy for the data item according to the data type of the data item; Analyzing the access pattern of the data item according to the access pattern analysis strategy of the data item and based on the data feature information of the data item; The data types are divided into at least two types: structured data and unstructured data.

3. The method according to claim 2, characterized in that The step of determining, for any of the data items, an access pattern analysis strategy for the data item according to the data type of the data item includes: For any of the data items, when the data item is structured data, determining an access pattern analysis strategy for the data item according to a detailed type of the data item represented by data feature information of the data item; In the case where the data item is unstructured data, determining an access pattern analysis strategy for the data item according to a data format represented by data feature information of the data item; The detailed types of the data items are divided into at least two types: order data items and user information data items, and the data formats include image formats, video formats and audio formats.

4. The method according to claim 3, characterized in that The step of analyzing the access pattern of the data item according to the access pattern analysis strategy of the data item and based on the data feature information of the data item includes: For any of the data items, according to the access pattern analysis strategy of the data item, based on the current access frequency, creation time and modification time represented by the data feature information of the data item, predict the access frequency of the data item in a preset future period, and obtain the future access frequency prediction result of the data item; An access pattern of the data item is determined according to a prediction result of the future access frequency of the data item.

5. The method according to claim 1, characterized in that Determining a target data cleaning strategy for the storage system according to a current business mode of the storage system and an access mode of each of the data items includes: screening low-frequency access data items in the storage system according to the access pattern of each of the data items; Analyzing the access cycle information of each of the infrequently accessed data items to determine the last access time of each of the infrequently accessed data items; According to the last access time of each of the low-frequency access data items, the low-frequency access data items are divided into short-term low-frequency access data items and long-term low-frequency access data items; In the case where the current business mode of the storage system is the normal mode, the short-term low-frequency access data items are used as data items to be deleted, and the long-term low-frequency access data items are used as data items to be migrated; Determining a target data cleaning strategy of the storage system is to delete the data items to be deleted and to migrate the data items to be migrated; In a case where the current business mode of the storage system is a high availability mode, the short-term low-frequency access data items are used as data items to be migrated, and the long-term low-frequency access data items are used as data items to be compressed; Determining a target data cleaning strategy of the storage system is to perform storage medium migration processing on the data items to be migrated and to perform data compression processing on the data items to be compressed; The last access time of the short-term infrequently accessed data item is later than the last access time of the long-term infrequently accessed data item.

6. The method according to claim 5, characterized in that The method further comprises: Before deleting the data item to be deleted, generating snapshot data of the data item to be deleted; In the event that a data item is accidentally deleted, the data item is restored based on the snapshot data.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining actual operation status and storage space usage of the storage system; Determine a data cleaning optimization strategy based on the actual operation of the storage system and the storage space usage; According to the data cleaning optimization strategy, the target data cleaning strategy is optimized.

8. A data cleaning device, characterized in that: include: An acquisition module, used to acquire data feature information of each data item in the storage system, a data type of each data item and a current business mode of the storage system; A first determination module, configured to determine, for any of the data items, an access mode of the data item according to data feature information and a data type of the data item; A second determination module, configured to determine a target data cleaning strategy for the storage system according to a current business mode of the storage system and an access mode of each of the data items; The cleaning module is used to clean up data in the storage system according to the target data cleaning strategy of the storage system.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

Images