Dynamic storage management using predictive analysis

Abstract

Systems and methods are disclosed to collect file data for log files associated with a distributed file system, the file data including size values and time values associated with the log files, segment the log files based on the time values to create time-based partitions, identify historical patterns in the file data, and optimize a data retention policy for the distributed file system based on the historical patterns.

Description

Technical Field

[0001] The following disclosures generally relate to storage management, and more specifically, to systems and methods for optimizing data retention strategies. Summary of the Invention

[0002] In one aspect, this disclosure provides a computer-implemented method comprising: collecting file data of a log file associated with a distributed file system by a data storage management system, the file data including a size value and a time value associated with the log file; segmenting the log file based on the time value by the data storage management system to create time-based partitions; identifying historical patterns in the file data by the data storage management system; and optimizing a data retention strategy of the distributed file system based on the historical patterns by the data storage management system.

[0003] In one aspect, this disclosure provides a data storage management system for optimizing a data retention strategy of a distributed file system. The data storage management system includes a data processing module configured to: retrieve file data of a log file associated with the distributed file system, the file data including a size value and a time value associated with the log file; and segment the log file based on the time value to create time-based partitions. The data storage management system may further include a prediction module configured to communicate with the data processing module, wherein the prediction module is configured to identify historical patterns in the file data. The data storage management system may further include an optimizer module configured to communicate with the prediction module, wherein the optimizer module is configured to optimize the data retention strategy of the distributed file system based on the historical patterns. Attached Figure Description

[0004] In this description, specific details, such as particular aspects, procedures, and techniques, are set forth for purposes of explanation and not limitation, in order to provide a thorough understanding of the art. However, those skilled in the art will understand that the art can be practiced in other ways, departing from these specific details.

[0005] The accompanying drawings and the following detailed description are incorporated in and form part of this specification, and are used to further illustrate aspects of the claimed disclosure and to explain the various principles and advantages of those aspects. In the drawings, the same reference numerals in the various views refer to the same or functionally similar elements.

[0006] The devices, systems, and methods disclosed herein have been represented by conventional symbols in the accompanying drawings, showing only those specific details relevant to understanding various aspects of this disclosure, so as not to obscure this disclosure by details that are obvious to those skilled in the art who benefit from the description herein.

[0007] Figure 1 A system for managing storage in a distributed file system, according to at least one aspect of this disclosure, is shown.

[0008] Figure 2 A method for managing storage in a distributed file system according to at least one aspect of this disclosure is shown.

[0009] Figure 3 A method for identifying historical patterns in file data of a distributed file system is shown, according to at least one aspect of this disclosure.

[0010] Figure 4 A method for predicting data retention strategies for distributed file systems according to at least one aspect of this disclosure is shown.

[0011] Figure 5 It is an optimized method for implementing an associated data retention strategy according to at least one aspect of this disclosure.

[0012] Figure 6 It is a block diagram of a computer device having a data processing subsystem or component according to at least one aspect of the present disclosure.

[0013] Figure 7 This is a schematic representation of an example system including a host according to at least one aspect of this disclosure, within which a set of instructions for performing any one or more of the methods discussed herein can be executed. Detailed Implementation

[0014] The following disclosure provides exemplary systems, apparatuses, and methods for conducting financial transactions and related activities. While references to such financial transactions may be made in the examples provided below, the scope is not limited thereto. That is, the systems, methods, and apparatuses described can be used for any suitable purpose.

[0015] Before discussing specific embodiments, aspects, or examples, the following provides some descriptions of the terms used herein.

[0016] As used herein, the term “system” may refer to one or more computing devices or a combination of computing devices (e.g., processor, server, client device, software application, components of such computing devices, etc.).

[0017] The surge in data has brought numerous challenges to data storage and management. Large data volumes require powerful hardware, efficient algorithms, and scalable systems to effectively process and analyze them. Therefore, distributed systems such as the Hadoop Distributed File System (HDFS) and cloud-based solutions have become crucial. These systems distribute data across multiple nodes, ensuring fault tolerance and high availability. However, managing and maintaining distributed file systems can be complex and requires expertise. Furthermore, increasing storage capacity is associated with increased costs, leading to the development of data retention strategies that determine appropriate data retention periods.

[0018] Organizations that manually manage data retention, have static data retention policies, or manually update their data retention policies spend significant resources defining what data should be retained, for how long, and under what circumstances it should be deleted—even though manual analysis of data usage and access patterns can be time-consuming and error-prone. Various aspects of this disclosure provide methods and systems for dynamic data storage that employ predictive analytics to address many of these challenges.

[0019] This disclosure provides a method 100 for dynamically optimizing data retention strategies in a data storage system. Figure 2 ) and System 50 ( Figure 1 For the sake of brevity, the following descriptions of Method 100 and System 50 focus on HDFS storage transaction files, but it will be readily understood that the descriptions associated with Method 100 and System 50 can be readily applied to any suitable data storage system and any suitable data file associated with any file type that can be stored on a file storage system.

[0020] System 50 includes a data processing module 51, a machine learning module 52, and an optimizer module 54. The components of system 50 can be implemented using software, hardware, or a combination of both. Figure 5 and Figure 6 Computer devices and systems including a host are depicted separately, which can be used individually or in combination to implement one or more components, such as system 50. While system 50 includes three components, it is readily understood that it can include more or fewer than the three described components. For example, system 50 may include only data processing module 51 and machine learning module 52, but not, for example, optimizer module 54. Furthermore, a configuration belonging to one module may be implemented or partially implemented by one of the other two components or additional components.

[0021] System 50 interacts with one or more distributed file systems to optimize data retention strategies associated with these systems. Figure 1The communication link 56 between the data processing module 51 and HDFS 55 is depicted. Additional communication links can connect the data processing module 51 to other distributed file systems. The data processing module 51 collects and preprocesses file data from log files stored in HDFS 55 to dynamically optimize its data retention strategy. The machine learning module 52 analyzes file data associated with log files on HDFS 55 to dynamically and autonomously make decisions regarding the retention of log files in HDFS 55.

[0022] The optimizer module 54 can also be connected to the one or more distributed file systems via a dedicated communication link or a shared communication link with the data processing module 51. Figure 1 A communication link 57 is depicted that interconnects the optimizer module 54 with HDFS 55 to implement or at least transmit optimization decisions. The optimizer module 54 monitors and adjusts retention policies based on data patterns of changes in collected data files analyzed by the machine learning module 52, thereby ensuring efficient storage management over time.

[0023] System 50 may also include a user interface 58. User interface 58 may include a dashboard to visualize historical data patterns and retention policies associated with the distributed file system managed by system 50.

[0024] Go to Figure 2 Method 100 is a computer-implemented method for optimizing a data retention strategy in a distributed file system. Method 100 may be implemented wholly or partially by system 50. Method 100 includes collecting 101 file data from log files associated with a distributed file system, such as HDFS 55. Data processing module 51 can access and read the file data stored on HDFS 55. The file data may include, for example, metadata associated with the log files stored on HDFS 55. The file data may include, for example, file size and / or file creation data, such as creation month, creation day, creation hour, and / or any suitable creation time. In some aspects, data processing module 51 utilizes communication link 56 to access data files stored on HDFS 55 and retrieve the data files for further processing by system 50.

[0025] Method 100 also includes splitting the log file 102 based on time values ​​in the file data to create time-based partitions. Data processing module 51 splits the log file by creation year, creation month, and / or creation day based on the retrieved file data. Data processing module 51 then determines the total size of the log files grouped into each partition. Therefore, data processing module 51 outputs multiple partitions characterized by time values ​​associated with the creation of the log files in HDFS 55.

[0026] Method 100 also includes identifying historical patterns in the file data associated with the distributed file system. The output of the data processing module 51 forms the input to the machine learning module 52. The log files are grouped by the machine learning module 52 in various ways to identify historical patterns associated with data retention in HDFS 55. The machine learning module 52 applies clustering algorithms, such as K-means clustering, to identify historical patterns associated with the file data.

[0027] Historical patterns can indicate periods of high, low, and / or normal utilization. In some respects, historical patterns indicate transitions, such as from high to low utilization or from low to high utilization. The machine learning module 52 can also analyze historical retention policies and their impact on HDFS 55. For example, different weights can be assigned to this information depending on the impact of such policies on HDFS 55's performance. Other factors, such as the frequency of log file access, can also be considered. In one example, the machine learning module 52 analyzes historical (e.g., the past four to five years) log generation rates to identify historical patterns that indicate past performance of HDFS 55.

[0028] Figure 3 A method 200 for identifying historical patterns in file data is illustrated, which can be performed by a machine learning module 52. Method 200 includes grouping the file data 201 into groups characterized by a time parameter (e.g., the month of file creation) representing the file creation time value. Additionally, method 200 includes determining 202 an average size associated with the group based on the collected file sizes of log files within the group, and the method may further include identifying 203 historical patterns based on the determined average size associated with the group and based on the time parameter associated with the group.

[0029] In one example, identifying 103 historical patterns involves using a K-means clustering algorithm to identify multiple clusters based on the determined average size associated with the clusters and based on time parameters associated with the clusters. A retention time is assigned to each cluster based on the identified 103 historical patterns. For example, a regular operation cluster might be associated with a log generation parameter (e.g., rate, such as the number of log files per day or the total file size per day) L1, and it could be assigned a retention time T1. A minor system update cluster might be associated with a log generation parameter (e.g., rate, such as the number of log files per day or the total file size per day) L2 greater than L1, and it could be assigned a retention time T2 less than T1. Similarly, a major system update cluster might be associated with a log generation parameter (e.g., rate, such as the number of log files per day or the total file size per day) L3 greater than L2, and it could be assigned a retention time T3 less than T2. ​​Finally, a peak user activity cluster might be associated with a log generation parameter (e.g., rate, such as the number of log files per day or the total file size per day) L4 greater than L3, and it could be assigned a retention time T4 less than T3. It will be easy to understand that the clusters listed are for illustrative purposes only, and different, more or fewer clusters may be identified.

[0030] Refer again Figure 2 Method 100 also includes optimizing the data retention strategy for the 104 log files based on historical patterns. In one example, optimizing the 104 data retention strategy could include predicting the data retention strategy for the log files in the current time unit (e.g., the current month). Figure 4 As shown, method 300 predicts a data retention strategy by grouping log files into groups based on a shared time parameter 301, for example, grouping log files by day, and determining 302 log generation parameters for each group (e.g., the number of log files per day or the total size per day). Method 300 also includes matching each group to a cluster based on a comparison of the log generation parameters with the group's log generation parameters, for example, by matching each group to a cluster with a log generation parameter value that is closest to the group's log generation parameter value. Then, a data retention period 304 is assigned to the group based on the matched clusters.

[0031] In one example, log generation parameters L1, L2, L3, and L4 represent the log generation rate (e.g., the total log file size per day). Therefore, new / current log files are grouped by day, and the total log file size per day is calculated for each group. Each group is then matched to one of the multiple clusters based on the group's log generation rate and the cluster's log generation rate. A retention policy is then assigned to each group based on the cluster assignment. For example, the first group with the log generation rate closest to L4 is matched to the cluster with peak user activity and is therefore assigned a retention time of T4.

[0032] Alternatively, according to method 100, optimizing the data retention policy 104 may include modifying the previously determined data retention period associated with older log files. For example, a group of log files previously matched to one cluster may be switched to another cluster by the machine learning module 52, resulting in a change in the group's retention period assignment. Therefore, system 50 dynamically manages the data retention policy associated with HDFS 55. The data retention policy associated with the current / new log files is dynamically determined based on past behavior. Furthermore, the data retention policy previously set for older log files can be dynamically modified to ensure optimal operation of HDFS 55.

[0033] Depending on the selected settings, system 50 can automatically implement data retention policy optimization 104 or request user input. Figure 5 This is a method 400 that can be executed by system 50 to optimize the data retention policy associated with HDFS 55. Method 400 includes prediction 401 for the data retention policy of HDFS 55. If automatic tuning 402 is selected, method 400 automatically tunes the data retention policy 403 based, for example, on a matching cluster. However, if automatic tuning 402 is not selected, method 400 issues an alert 404 to notify the administrator of the proposed change to the retention policy and awaits approval for implementation.

[0034] Method 400 can be executed by system 50. In one example, optimizer module 54 issues a 403 warning and alerts the administrator via user interface 58 regarding suggested changes to the retention policy. Alternatively, optimizer module 54 can trigger alerts, such as alerts based on the total storage capacity of HDFS 55. Figure 5 As shown, method 400 may further include detecting 405 whether a storage threshold has been triggered.

[0035] In one example, a storage threshold is triggered if an implementation of a recommended adjustment based on a data retention policy reaches or exceeds, or is about to reach or exceed, a storage threshold. For example, the storage threshold could be a predefined percentage of the total storage capacity of HDFS 55.

[0036] If a storage threshold is triggered and auto-adjustment is selected (406), the HDFS 55 storage level is adjusted (407). For example, adjustments may include deleting older and / or unnecessary data. However, if auto-adjustment (406) is not selected, a 404 alert is issued to notify the administrator that the storage threshold has been reached or exceeded.

[0037] User interface 58 may include a dashboard displaying daily log file creation rate, data policy forecasts, current storage usage, and / or applied retention policies (whether automatically tuned or manually approved). This provides a visual representation of storage status and implemented policies.

[0038] The optimizer module 54 can further solicit user feedback regarding the optimization of the determined data retention strategy. User input can be received through the user interface 58. For example, the optimizer module 54 can transmit user feedback to the machine learning module 52 to update historical data and the retention strategy.

[0039] As mentioned above Figures 1 to 4 Each of the aforementioned systems and methods described herein may include or utilize multiple computer devices, computer systems, etc. In other words, to utilize the systems and methods disclosed herein, at least one of a computer device, computer system, or the like may be implemented. The following section discusses… Figure 6 The computer device 3000 shown in the image and Figure 7 The example system 4000 shown herein describes each of these computer devices, computer systems, etc. in more detail, providing a connection between the solutions disclosed herein and how such solutions can be implemented within commercial entities such as payment networks, processing networks, and payment processing networks.

[0040] Figure 6 It is a block diagram of a computer device 3000 having a data processing subsystem or component according to at least one aspect of this disclosure. Figure 6The subsystems shown are interconnected via system bus 3010. Additional subsystems are shown, such as printer 3018, keyboard 3026, fixed disk 3028 (or other memory including computer-readable media), monitor 3022 (coupled to display adapter 3020), etc. Peripheral devices and input / output (I / O) devices coupled to I / O controller 3012 (which may be a processor or other suitable controller) can be connected to the computer system via any number of means known in the art (e.g., serial port 3024). For example, serial port 3024 or external interface 3030 can be used to connect the computer device to a wide area network (e.g., the Internet), a mouse input device, or a scanner. The interconnection via system bus 3010 allows central processing unit 3016 to communicate with each subsystem and control the execution of instructions from system memory 3014 or fixed disk 3028, as well as the exchange of information between subsystems. System memory 3014 and / or fixed disk 3028 may be embodied in computer-readable media.

[0041] Figure 7 This is a schematic representation of an example system 4000 including a host 4002, according to at least one aspect of this disclosure, within which a set of instructions for performing any one or more of the methods discussed herein can be executed. In various aspects, the host 4002 operates as a standalone device or can be connected (e.g., networked) to other machines. In a network deployment, the host 4002 can operate as a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The host 4002 may be a computer or computing device; a personal computer (PC); a tablet PC; a set-top box; a personal digital assistant; a cellular phone; a portable music player (e.g., a portable hard disk audio device, such as a Moving Picture Experts Group Audio Layer 3 (MP3) player); a network appliance; a network router, switch, or bridge; or any machine capable of executing a set of instructions (sequentially or otherwise) specifying the actions to be taken by the machine. Further, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or collectively execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0042] Example system 4000 includes a host 4002 that runs a host operating system (OS) 4004 on one or more processors / processor cores 4006 (e.g., a central processing unit (CPU), a graphics processing unit, or both) and various memory nodes 4008. The host OS 4004 may include a super manager 4010 capable of control functions and / or communicating with virtual machines (VMs) 4012 running on machine-readable media. VM 4012 may also include a virtual CPU or vCPU 4014. Memory nodes 4008 may be linked or pinned to virtual memory nodes or vNodes 4016. When a memory node 4008 is linked or pinned to a corresponding vNode 4016, data can subsequently be directly mapped from the memory node 4008 to its corresponding vNode 4016.

[0043] All the various components shown in host 4002 can be connected to and linked to each other, or communicate with each other via a bus (not shown) or through other coupling or communication channels or mechanisms. Host 4002 may also include a video display, audio device or other peripheral device 4018 (e.g., liquid crystal display, alphanumeric input device (including, for example, a keyboard), cursor control device (e.g., mouse), voice recognition or biometric authentication unit, external driver, signal generation device (e.g., speaker)), persistent storage device 4020 (also referred to as a disk drive unit), and network interface device 4022. Host 4002 may also include a data encryption module (not shown) for encrypting data. The components disposed in host 4002 are components commonly found in computer systems that can be adapted for use with aspects of this disclosure, and are intended to represent a broad category of such computer components known in the art. Thus, system 4000 may be a server, minicomputer, host computer, or any other computer system. Computers may also include different bus configurations, networking platforms, multiprocessor platforms, etc. It can use various operating systems, including UNIX, LINUX, WINDOWS, QNX ANDROID, IOS, CHROME, TIZEN and other suitable operating systems.

[0044] The disk drive unit 4024 may also be a solid-state drive, a hard disk drive, or other drive including computer or machine-readable media on which one or more sets of instructions and data structures (e.g., data / instructions 4026) are stored, which embody or utilize any one or more methods or functions described herein. The data / instructions 4026 may also reside wholly or at least partially within main memory node 4008 and / or processor 4006 during execution by host 4002. The data / instructions 4026 may be further sent or received via network 4028 via network interface device 4022 utilizing any of several well-known transport protocols (e.g., Hypertext Transfer Protocol (HTTP)).

[0045] Processor 4006 and memory node 4008 may also include machine-readable media. The term "computer-readable media" or "machine-readable media" should be considered as including a single or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions. The term "computer-readable media" should also be considered as including any medium capable of storing, encoding, or carrying a set of instructions for execution by host 4002 and causing host 4002 to perform any or more methods of this application, or any medium capable of storing, encoding, or carrying data structures utilized by or associated with such instruction set. Therefore, the term "computer-readable media" should be understood to include, but is not limited to, solid-state memory, optical and magnetic media, and carrier signals. Such media may also include, but is not limited to, hard disks, floppy disks, flash memory cards, digital video optical discs, random access memory (RAM), read-only memory (ROM), etc. The exemplary aspects described herein may be implemented in an operating environment including software installed on a computer, in hardware, or in a combination of software and hardware.

[0046] Those skilled in the art will recognize that an Internet service can be configured to provide Internet access to one or more computing devices coupled to the Internet service, and the computing devices may include one or more processors, buses, memory devices, display devices, I / O devices, etc. Furthermore, those skilled in the art will understand that the Internet service can be coupled to one or more databases, repositories, servers, etc., which can be used to implement any aspect of the various aspects of this disclosure as described herein.

[0047] Computer program instructions may also be loaded onto a computer, server, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0048] Suitable networks may include any one or more of the following, or connections to any one or more of the following: local intranet; personal area network (PAN); local area network (LAN); wide area network (WAN); metropolitan area network (MAN); virtual private network (VPN); storage area network (SAN); frame relay connection; advanced intelligent network (AIN) connection; synchronous fiber optic network (SONET) connection; digital T1, T3, E1, or E3 line; digital data service (DDS) connection; digital subscriber line (DSL) connection; Ethernet connection; integrated services digital network (ISDN) line; dial-up port, such as V.90, V.34, or V.34bis analog modem connection; cable modem; asynchronous transfer mode (ATM) connection; or fiber distributed data interface (FDDI) or copper distributed data interface (CDDI) connection. Furthermore, communications may include links to any of a variety of wireless networks, including Wireless Application Protocol (WAP), General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA) or Time Division Multiple Access (TDMA), cellular telephone networks, Global Positioning System (GPS), Cellular Digital Packet Data (CDPD), Dynamic Research Inc. (RIM) full-duplex paging network, Bluetooth radio, or IEEE 802.11-based radio frequency (RF) networks. Network 4028 may also include RS-232 serial connections, IEEE-1394 (FireWire) connections, Fibre Channel connections, IrDA (Infrared (IR)) ports, Small Computer System Interface (SCSI) connections, Universal Serial Bus (USB) connections, or other wired or wireless, digital or analog interfaces or connections, mesh or Digi. ® Any one or more connected to or interfacing with the network.

[0049] Generally, a cloud-based computing environment is a resource that typically combines large groups of processors (such as within a web server) with computing power and / or large groups of computer memory or storage devices with storage capacity. Systems providing cloud-based resources may be available only to their owners, or such systems may be accessible to external users who deploy applications within the computing infrastructure to benefit from large computing or storage resources.

[0050] For example, a cloud is formed by a network of network servers comprising multiple computing devices (such as host 4002), where each server 4030 (or at least several) provides processor and / or storage resources. These servers manage workloads provided by multiple users (e.g., cloud resource customers or other users). Typically, each user's workload requirements for the cloud change in real time, sometimes dramatically. The nature and extent of these changes usually depend on the type of business associated with the user.

[0051] It is worth noting that any hardware platform suitable for performing the processes described herein is suitable for use with the technology. As used herein, the terms "computer-readable storage medium" and "computer-readable storage media" refer to any one or more media that participate in providing instructions to the CPU for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks, such as fixed disks. Volatile media include dynamic memory, such as system RAM. Transmission media include coaxial cables, copper wires, and optical fibers, which include conductors comprising one side of a bus. Transmission media can also take the form of acoustic or optical waves, such as acoustic or optical waves generated during RF and IR data communications. Common forms of computer-readable media include, for example, floppy disks, hard disks, magnetic tapes, any other magnetic media, optical disc ROMs (CD-ROMs), digital video discs, any other optical media, any other physical media with markings or perforations, RAM, programmable ROMs, erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), flash memory EPROMs, any other memory chips or data exchange adapters, carrier waves, or any other media that a computer can read.

[0052] Various forms of computer-readable media can participate in carrying one or more sequences of one or more instructions to the CPU for execution. A bus carries data to system RAM, from which the CPU retrieves and executes instructions. Instructions received from system RAM may optionally be stored on a disk before or after execution by the CPU.

[0053] Computer program code used to perform operations on aspects of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language, Go, Python, or other programming languages ​​including assembly language. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including LANs or WANs, or can connect to an external computer (e.g., via the Internet using an Internet service provider).

[0054] Examples of the methods and systems described in accordance with various aspects of this disclosure are provided below in the numbered clauses. Any aspect of a system or method may include any one or more of the numbered clauses described below, and any combination thereof.

[0055] Clause 1. A computer-implemented method comprising: collecting file data of a log file associated with a distributed file system by a data storage management system, the file data including a size value and a time value associated with the log file; segmenting the log file based on the time value by the data storage management system to create time-based partitions; identifying historical patterns in the file data by the data storage management system; and optimizing a data retention strategy of the distributed file system based on the historical patterns by the data storage management system.

[0056] Clause 2. The computer-implemented method according to Clause 1, wherein the optimization of the data retention policy includes adjusting the data retention policy for a subset of the log files.

[0057] Clause 3. A computer-implemented method according to any one of Clauses 2 to 3, wherein collecting the file data of the log file includes reading the metadata of the log file on the distributed file system.

[0058] Clause 4. A computer-implemented method according to any one of Clauses 2 to 4, wherein the time value includes the file generation time value of the log file.

[0059] Clause 5. A computer-implemented method according to Clause 4, wherein identifying the historical pattern comprises: grouping the file data into groups characterized by a time parameter, wherein the grouping of the file data is based on the file generation time value; and determining an average size associated with the group, wherein the average size is based on the size value.

[0060] Clause 6. The computer-implemented method according to Clause 5, wherein identifying the historical pattern is based on the average size and the time parameter.

[0061] Clause 7. The computer-implemented method according to Clause 6, wherein identifying the historical pattern is also based on the file access frequency of the log file.

[0062] Clause 8. A computer-implemented method according to any one of Clauses 2 to 5, wherein grouping the file data into groups characterized by the time parameter comprises grouping the file data by month based on the file generation time value.

[0063] Clause 9. A computer-implemented method according to any one of Clauses 2 to 8, wherein identifying the historical pattern further comprises: applying a clustering algorithm to the file data to identify multiple clusters based on the file data.

[0064] Clause 10. A data storage management system for optimizing a data retention strategy of a distributed file system, the data storage management system comprising: a data processing module configured to: retrieve file data of a log file associated with the distributed file system, the file data including a size value and a time value associated with the log file; and segment the log file based on the time value to create time-based partitions; a prediction module configured to communicate with the data processing module, wherein the prediction module is configured to identify historical patterns in the file data; and an optimizer module configured to communicate with the prediction module, wherein the optimizer module is configured to optimize the data retention strategy of the distributed file system based on the historical patterns.

[0065] Clause 11. The data storage management system according to Clause 10, wherein the prediction module is configured to: group the file data into groups characterized by time parameters based on the time value; and determine an average size associated with the group, wherein the average size is based on the size value.

[0066] Clause 12. The data storage management system according to Clause 11, wherein the prediction module is used to identify the historical pattern based on the average size and the time parameter.

[0067] Clause 13. The data storage management system pursuant to Clause 12, wherein the prediction module is used to further identify the historical patterns based on the file access frequency of the log files.

[0068] Clause 14. A data storage management system according to any one of Clauses 10 to 14, wherein the prediction module is used to identify the historical pattern by applying a clustering algorithm to the file data to identify multiple clusters based on the file data.

[0069] Clause 15. The data storage management system pursuant to Clause 14, wherein the plurality of clusters includes at least one of a regular operation cluster, a minor system update cluster, a major system update cluster, or a peak user activity cluster.

[0070] Clause 16. The data storage management system as described in Clause 14, wherein each of the plurality of clusters is assigned a log generation rate and a retention period.

[0071] Clause 17. A data storage management system according to any one of Clauses 10 to 16, wherein the optimizer module is configured to optimize the data retention policy of the distributed file system by predicting the data retention policy associated with a newer subset of the log files based on the historical patterns associated with older subsets of the log files.

[0072] Clause 18. The data storage management system according to Clause 17, wherein the optimizer module is configured to: determine a log generation rate associated with the newer subset of the log files; assign the new file data to one of the plurality of clusters based on the log generation rate; and automatically adjust the data retention of the newer subset of the log files based on the assigned cluster among the plurality of clusters.

[0073] Clause 19. The computer-implemented method according to Clause 14 further includes a user interface for communicating with the optimizer module, wherein the optimizer module is configured to output an alarm signal so that the user interface issues an alarm based on an assigned cluster among the plurality of clusters.

[0074] Clause 20. The computer-implemented method of claim 19, wherein the user interface is configured to: receive user input to adjust the data retention policy; and transmit an output signal to the optimizer module based on the user input, the output signal being configured to cause the optimizer module to adjust the optimization of the data retention policy, the optimization being based on the historical pattern.

[0075] The foregoing detailed description has illustrated various forms of systems and / or processes using block diagrams, flowcharts, and / or examples. Where such block diagrams, flowcharts, and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within such block diagrams, flowcharts, and / or examples can be implemented individually and / or collectively by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be implemented, in whole or in part, equivalently in an integrated circuit as one or more computer programs (e.g., one or more programs running on one or more computer systems), one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), firmware, or virtually any combination thereof, and that designing circuit systems and / or writing code for software and / or firmware according to this disclosure will be entirely within the skill of those skilled in the art. Furthermore, those skilled in the art will understand that the mechanisms of the subject matter described herein are capable of being distributed in various forms as one or more program products, and that the illustrative forms of the subject matter described herein apply regardless of the specific type of signal-bearing medium used to actually perform said distribution.

[0076] Instructions for programming logic to execute various disclosed aspects may be stored in the system's memory, such as dynamic RAM, cache, flash memory, or other storage devices. Furthermore, the instructions may be distributed via a network or by means of other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAM, EPROMs, EEPROMs, magnetic cards or optical cards, flash memory, or tangible machine-readable storage devices for transmitting information via the Internet through electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, IR signals, digital signals). Therefore, non-transitory computer-readable media includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0077] Any software component or function described in this application can be implemented as software code executed by a processor using any suitable computer language such as Python, Java, C++, or Perl, using techniques such as conventional or object-oriented methods. The software code can be stored as a series of instructions or commands on a computer-readable medium (such as RAM, ROM), a magnetic medium (such as a hard disk drive or floppy disk), or an optical medium (such as a CD-ROM). Any such computer-readable medium can reside on or within a single computing device, and can exist on different computing devices or within a system or network.

[0078] As used in any aspect of this document, the term "logic" can refer to an application, software, firmware, and / or circuit system configured to perform any of the foregoing operations. Software can be embodied as software packages, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware can be embodied as hard-coded (e.g., non-volatile) code, instructions, instruction sets, and / or data in a memory device.

[0079] As used in any aspect of this document, the terms “component,” “system,” “module,” etc., may refer to a computer-related entity, or hardware, a combination of hardware and software, software, or software in execution.

[0080] As used in any aspect of this document, "algorithm" refers to a self-consistent sequence of steps that produces a desired result, where "step" refers to the manipulation of physical quantities and / or logical states, which may (but do not necessarily need to) take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. Common usage refers to these signals as bits, values, elements, symbols, characters, terms, numbers, etc. These terms and similar terms may be associated with appropriate physical quantities and are merely convenient labels applied to these quantities and / or states.

[0081] The network may include a packet-switched network. Communication devices may be able to communicate with each other using selected packet-switched network communication protocols. An example communication protocol may include an Ethernet communication protocol that may allow communication using Transmission Control Protocol / Internet Protocol. The Ethernet protocol may conform to or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" published by IEEE in December 2008 and / or subsequent versions of this standard. Alternatively or additionally, communication devices may be able to communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards issued by the International Telecommunication Union-Telecommunication Standardization Sector. Alternatively or additionally, communication devices may be able to communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol may conform to or be compatible with standards issued by the Consultative Committee for International Telegraph and Telephone and / or the American National Standards Institute. Alternatively or additionally, transceivers may be able to communicate with each other using the ATM communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard entitled "ATM-MPLS Network Interworking 2.0" and / or subsequent versions of this standard, published by the ATM Forum in August 2001. Of course, this document also considers different and / or later-developed connection-oriented network communication protocols.

[0082] Unless explicitly stated in the foregoing disclosure, it should be understood that throughout this disclosure, the use of terms such as “processing,” “computing,” “operation,” “determining,” “displaying,” etc., refers to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in computer system registers and memories and transform them into other data similarly represented as physical quantities in computer system memories or registers or other such information storage, transmission, or display devices.

[0083] One or more components may be referred to herein as “configured to,” “configurable to,” “operable to,” “suitable for,” “capable of,” “compliant with,” etc. Unless the context otherwise requires, those skilled in the art will recognize that “configured to” can generally encompass active components, inactive components, and / or standby components.

[0084] Those skilled in the art will recognize that, generally, the terminology used herein, and especially in the appended claims (e.g., the body of the appended claims), is intended to be “open-ended” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “including but not limited to,” etc.). Those skilled in the art will further understand that if the intent is a particular number of introduced claim statements, such intent will be expressly stated in the claims, and without such statements, such intent does not exist. For example, to aid understanding, the following appended claims may contain the introductory phrases “at least one” and “one or more” to introduce claim statements. However, the use of such phrases should not be construed as implying that introducing a claim statement with the indefinite article "a" or "an" limits any particular claim containing such an introduced claim statement to a claim containing only one such statement, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); the same applies to the use of definite articles used to introduce a claim statement.

[0085] Furthermore, even when a specific number is explicitly stated in the introduced claims, those skilled in the art will recognize that such a statement should generally be interpreted as meaning at least the stated number (e.g., the simple statement "two statements" without other modifiers generally means at least two statements, or two or more statements). Moreover, in these cases where conventions such as "at least one of A, B, and C" are used, such constructions are generally intended to be in the sense that those skilled in the art would understand from the convention (e.g., "a system having at least one of A, B, and C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In these cases where conventions such as "at least one of A, B, or C" are used, such constructions are generally intended to be in the sense that those skilled in the art would understand from the convention (e.g., "a system having at least one of A, B, or C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art will further understand that, generally, unless the context otherwise indicates, separate words and / or phrases presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of including one term, either term, or both terms. For example, the phrase "A or B" will generally be understood to include the possibility of including "A", "B", or "A and B".

[0086] Regarding the appended claims, those skilled in the art will understand that the operations described herein can generally be performed in any order. Furthermore, although the various operation flowcharts are presented in a sequence, it should be understood that the various operations can be performed in any order other than those shown, or can be performed simultaneously. Unless the context otherwise requires, instances of such alternative ordering can include overlapping, interleaving, interruption, reordering, ascending, preparatory, supplementary, simultaneous, inverted, or other variations of ordering. Moreover, unless the context otherwise requires, terms such as "in response to," "related to," or other past tense adjectives are generally not intended to exclude such variations.

[0087] It is important to note that any reference to "one aspect," "an aspect," "an example," "a paradigm," etc., means that a particular feature, structure, or characteristic described in connection with said aspect is included in at least one aspect. Therefore, the phrases "in one aspect," "in one aspect," "in an example," and "in a paradigm" appearing throughout the specification do not necessarily all refer to the same aspect. Furthermore, a particular feature, structure, or characteristic may be combined in one or more aspects in any suitable manner.

[0088] As used herein, the singular forms “a,” “an,” and “the” include plural indicators unless the context clearly indicates otherwise.

[0089] Any patent applications, patents, non-patent publications, or other disclosures cited in this specification and / or listed in any application data sheets are incorporated herein by reference, provided that the incorporated material does not contradict this document. Thus, and to the extent necessary, disclosures expressly set forth herein supersede any conflicting material incorporated herein by reference. It is claimed that any material or portion thereof incorporated herein by reference that conflicts with existing definitions, statements, or other disclosures set forth herein will be incorporated only to the extent that the incorporated material does not conflict with existing disclosures. This is not an admission that they are prior art.

[0090] In summary, the numerous benefits arising from adopting the concepts described herein have been described. One or more forms of the foregoing description have been presented for illustrative and descriptive purposes. It is not intended to be exhaustive or limited to the precise forms disclosed. Modifications or variations are possible in light of the foregoing teachings. The aforementioned forms have been chosen and described to illustrate principles and practical applications, thereby enabling those skilled in the art to utilize the various forms and make various modifications suitable for the particular purpose contemplated. The overall scope is intended to be defined by the claims filed herein.

Claims

1. A computer-implemented method comprising: The data storage management system collects file data from log files associated with the distributed file system, the file data including size and time values ​​associated with the log files; The data storage management system segments the log file based on the time value to create time-based partitions; The historical patterns in the file data are identified by the data storage and management system; as well as The data storage management system optimizes the data retention strategy of the distributed file system based on the historical patterns.

2. The computer-implemented method of claim 1, wherein the optimization of the data retention policy includes adjusting the data retention policy for a subset of the log files.

3. The computer-implemented method of claim 2, wherein collecting the file data of the log file includes reading the metadata of the log file on the distributed file system.

4. The computer-implemented method according to claim 1, wherein the time value includes the file generation time value of the log file.

5. The computer-implemented method according to claim 4, wherein identifying the historical pattern comprises: The file data is grouped into groups characterized by a time parameter, wherein the grouping of the file data is based on the file generation time value; as well as Determine the average size associated with the group, wherein the average size is based on the size value.

6. The computer-implemented method of claim 5, wherein identifying the historical pattern is based on the average size and the time parameter.

7. The computer-implemented method of claim 6, wherein identifying the historical pattern is further based on the file access frequency of the log file.

8. The computer-implemented method of claim 5, wherein grouping the file data into groups characterized by the time parameter comprises grouping the file data by month based on the file generation time value.

9. The computer-implemented method of claim 1, wherein identifying the historical pattern further comprises: Clustering algorithms are applied to the file data to identify multiple clusters based on the file data.

10. A data storage management system for optimizing data retention strategies in a distributed file system, the data storage management system comprising: The data processing module is used to: Retrieve file data from log files associated with the distributed file system, the file data including size and time values ​​associated with the log files; as well as The log file is split based on the time value to create time-based partitions; A prediction module, which communicates with the data processing module, wherein the prediction module identifies historical patterns in the file data; as well as An optimizer module, which communicates with the prediction module, wherein the optimizer module optimizes the data retention strategy of the distributed file system based on the historical pattern.

11. The data storage management system according to claim 10, wherein the prediction module is used to: The file data is grouped into groups characterized by time parameters based on the time values; and Determine the average size associated with the group, wherein the average size is based on the size value.

12. The data storage management system of claim 11, wherein the prediction module is used to identify the historical pattern based on the average size and the time parameter.

13. The data storage management system of claim 12, wherein the prediction module is used to further identify the historical pattern based on the file access frequency of the log file.

14. The data storage management system of claim 10, wherein the prediction module is used to identify the historical pattern by applying a clustering algorithm to the file data to identify multiple clusters based on the file data.

15. The data storage management system according to claim 14, wherein the plurality of clusters includes at least one of a regular operation cluster, a minor system update cluster, a major system update cluster, or a peak user activity cluster.

16. The data storage management system of claim 14, wherein each of the plurality of clusters is assigned a log generation rate and a retention period.

17. The data storage management system of claim 10, wherein the optimizer module is configured to optimize the data retention policy of the distributed file system by predicting the data retention policy associated with a newer subset of the log files based on the historical patterns associated with older subsets of the log files.

18. The data storage management system according to claim 17, wherein the optimizer module is used to: Determine the log generation rate associated with the newer subset of the log files; The new file data is assigned to one of the multiple clusters based on the log generation rate. as well as The data retention of the newer subset of the log files is automatically adjusted based on one of the assigned clusters among the plurality of clusters.

19. The data storage management system of claim 14, further comprising a user interface for communicating with the optimizer module, wherein the optimizer module is configured to output an alarm signal to cause the user interface to issue an alarm based on one of the assigned clusters among the plurality of clusters.

20. The data storage management system according to claim 19, wherein the user interface is used to: Receive user input to adjust the data retention policy; and Based on the user input, an output signal is transmitted to the optimizer module, which is used to enable the optimizer module to adjust the optimization of the data retention strategy, the optimization being based on the historical pattern.

Images