Generate and deform a collection of files in a folder / subfolder structure that collectively have the desired deduplication, compression, clustering, and commonality

Abstract

An exemplary method includes: receiving a set of file system parameters; creating a simulated file system based on the file system parameters; receiving a set of target characteristics of a file collection; based on the target characteristics, slicing a data stream into a set of data slices; populating a simulated file with the data slices to create the file collection; and morphing the file collection forward or backward from one generation to another without rewriting the entire file collection.

Description

[0001] Related Applications

[0002] This application relates to: U.S. Patent No. 10,038,733 entitled “GENERATING A LARGE, NON-COMPRESSIBLE DATA STREAM” issued on July 31, 2018 (Application No. 14 / 489,317, filed on September 17, 2014); U.S. Patent No. 10,114,832 entitled “GENERATING A DATA STREAM WITH APREDICTABLE CHANGE RATE” issued on October 30, 2018 (Application No. 14 / 489,363, filed on September 17, 2014); U.S. Patent No. 10,114,832 entitled “GENERATING A DATA STREAM WITH APREDICTABLE CHANGE RATE” issued on October 30, 2018 (Application No. 14 / 489,363, filed on September 17, 2014); U.S. Patent No. 10,114,832 entitled “GENERATING A DATA STREAM WITH APREDICTABLE CHANGE RATE” issued on October 30, 2018 10,114,850 (application number 14 / 489,295, filed on September 17, 2014); U.S. Patent US 10,235,134, entitled “ROTATING BIT VALUES WHILE GENERATING A LARGE, NON-COMPRESSIBLE DATA STREAM” issued on March 19, 2019 (application number 15 / 420,614, filed on January 31, 2017); and U.S. Patent US10,163,371, entitled “ROTATING BIT VALUES BASED ON A DATASTRUCTURE WHILE GENERATING A LARGE, NON-COMPRESSIBLE DATA STREAM” issued on December 25, 2018 (application number 15 / 420,633, filed on January 31, 2017). The present application also relates to: U.S. patent application entitled "GENERATING A DATA STREAM WITH CONFIGURABLE COMPRESSION" filed on the same day (application number 16 / 389,671); U.S. patent application entitled "GENERATING A DATA STREAM WITH CONFIGURABLE COMMONALITY" filed on the same day (application number 16 / 389,671); U.S. patent application entitled "GENERATING A DATA STREAM WITH CONFIGURABLE CHANGE RATE AND CLUSTERING CAPABILITY" filed on the same day (application number 16 / 389,729). The corresponding entire contents of all the aforementioned patents and applications are incorporated herein by reference. Technical Field

[0003] Embodiments of the present invention generally relate to the generation of data streams having various attributes. More specifically, at least some embodiments of the present invention relate to systems, hardware, software, computer-readable media, and methods for generating a collection of files based on a data stream having desired parameters. Background Art

[0004] Developers and others often need to simulate the data flow generated by an application that is in the development phase. Simulation of data flow enables developers to identify and correct any problems before the application is released, and to enhance the performance of the application.

[0005] Various algorithms have been developed to generate data streams. The data streams generated by these algorithms may have various desirable properties, but such data streams may be more suitable for use in some applications than in others. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to describe the manner in which at least some of the advantages and features of the present invention can be obtained, a more particular description of embodiments of the present invention will be presented by reference to specific embodiments of the present invention illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the present invention and are not therefore to be considered as limiting the scope thereof, and embodiments of the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0007] Figure 1 Aspects of an exemplary operating environment for some embodiments of the invention are disclosed.

[0008] Figure 2 Aspects of an exemplary file system composition module are disclosed.

[0009] Figure 3 Aspects of exemplary host configurations are disclosed.

[0010] Figure 4 Exemplary file systems and file structure schemes are disclosed.

[0011] Figure 5 An exemplary arrangement is disclosed in which a data stream is sliced ​​into files.

[0012] Figure 6 is a flow chart disclosing some general aspects of a method for creating a generation 0 file collection.

[0013] Figure 7 An exemplary arrangement is disclosed in which data streams are compared and a set of files is updated based on the comparison.

[0014] Figure 8is a flow chart disclosing aspects of an exemplary method for morphing a collection of files into different generations.

[0015] Fig. 9 is a flow chart disclosing aspects of an exemplary method for reverse morphing a collection of files into different generations. DETAILED DESCRIPTION

[0016] Embodiments of the present invention generally relate to the generation of data streams having various properties. More specifically, at least some embodiments of the present invention relate to systems, hardware, software, computer-readable media, and methods for generating a collection of files based on a data stream having desired characteristics.

[0017] In more detail, exemplary embodiments of the present invention may be employed in conjunction with various types of data streams, including, for example, data streams having particular characteristics. Such characteristics include, but are not limited to, for example, de-repeatability (i.e., the degree to which a data stream can be de-repeated, if any), compressibility, versatility, data change rate, and clustering. It should be noted that de-repeatability may exist within a single data stream and / or across multiple data streams. Systems and methods for generating such data streams are disclosed in related applications.

[0018] In conjunction with the file system with user-defined file structure, an exemplary embodiment of the present invention is implemented. More specifically, many parameters (such as the total size of a set of files, the average size of each entity in the set, the difference of entity size in the set, the distribution of set entities at each folder and subfolder, and the minimum, maximum, difference of each folder level) are collected and the parameters are used to create a simulated file structure of the file system. Then, each entity to be included in the file system is created in order, and the entity is filled with data from the data stream. When multiple data streams are adopted, it is not necessary to create entities in order.

[0019] In this way, various characteristics of one or more data streams are assigned to files in the file system. From another perspective, embodiments of the present invention generate and deform file sets in a file system having a specified folder / subfolder file structure configuration so that the resulting file groupings share characteristics that the client desires, such as, for example, deduplication, compression, clustering, and versatility. The following examples are illustrative.

[0020] The generation of a base file set may involve taking an L0 (also referred to as Day 0 in some contexts) version of a data stream having specific characteristics, and then using data from that data stream to populate simulation files to generate a base file set having those characteristics. However, embodiments of the present invention are not limited to generating base file sets.

[0021] For example, other embodiments of the present invention provide for transforming a file set forward or reversely into another version of the file set. This will be briefly described below by way of an example in which a file set is transformed forward from the N-1 generation to the N generation.

[0022] This particular deformation example involves generating Ln (day n) and Ln-1 (day n-1) versions of the data stream in memory, and then comparing these versions to each other in smaller blocks (e.g., such as 1KB). The size of the block can be selected as needed. For example, in an embodiment that simulates the operation of a tracker that tracks changes at the 64KB level, such as VM (VMware) CBT (changed block tracking), the size of the block can be 64KB. When a difference is identified, that is, when the Nth generation version of a block in a file is different from the N-1th generation version of the block, the N-1th generation version of the block will be overwritten with the Nth generation version of the block. Since the data generation rate can be very high, both the data stream comparison process and the file collection deformation process can be performed at high speed. In addition, since all existing entities are no longer read or written, only the affected files and affected areas are changed. As disclosed in more detail elsewhere herein and in related applications, file collection deformation can be performed without self-organizing data stream generation by using masks.

[0023] Advantageously, then, embodiments of the present invention can provide various benefits and improvements over the configuration and operation of conventional hardware, software, systems and methods. For example, embodiments of the present invention provide for high-speed generation of file collections having desired characteristics imparted by the data from which the file collections were generated. Embodiments of the present invention are particularly well suited for use in testing and evaluating backup applications and associated components and processes, as some backup applications are not configured to handle data streams, but can handle data in the form of file collections. In addition, embodiments of the present invention can be operated to deform file collections at the same or nearly the same speed as the data stream is generated. Embodiments of the present invention eliminate the need to copy entire files and / or collections, and instead write only the data that has changed.

[0024] It should be noted that the aforementioned advantages of various embodiments are presented by way of example only, and various other advantages of the exemplary embodiments of the present invention will become apparent from this disclosure. It should also be noted that not any embodiment can achieve or enable any such advantages disclosed herein.

[0025] A. Aspects of an Exemplary Operating Environment

[0026] The following is a discussion of various aspects of an exemplary operating environment for various embodiments of the present invention. This discussion is not intended to limit the scope of the present invention or the applicability of the embodiments in any way.

[0027] Typically, embodiments of the present invention can be implemented in conjunction with testing, analysis, diagnosis and evaluation of systems, software and components, which implement data generation and data management operations and / or cause the implementation of data generation and data management operations individually and / or together. Such data management operations may include, but are not limited to, data read / write / delete operations, data deduplication operations, data backup operations, data recovery operations, data cloning operations, data archiving operations and disaster recovery operations. Therefore, although the discussion herein may relate to the discussion of data protection environments and operations in some aspects, the scope of the present invention is not limited thereto. Then, more generally, the scope of the present invention includes any operating environment in which the disclosed concepts can be used. In some cases, embodiments of the present invention generate data streams for use in test systems and applications in various environments, an example of which is a data protection environment.

[0028] For example, the data protection environment can take the form of a public or private cloud storage environment, a local storage environment, and a hybrid storage environment including public and private elements, but the scope of the present invention also extends to any other type of data protection environment. Any of these exemplary storage environments can be partially or completely virtualized. The storage environment can include or consist of a data center that is operable to service read and write operations initiated by one or more clients.

[0029] In addition to the storage environment, the operating environment may also include one or more host devices, such as clients, each of which hosts one or more applications. Thus, a particular client may employ one or more instances of each of the one or more applications that generate data that is desired to be protected, or otherwise be associated with one or more instances. Typically, the applications employed by the client are not limited to any particular functionality or any type of functionality. For example, some exemplary applications and data include email applications (such as MS Exchange, file systems) and databases (such as Oracle databases and SQL server databases). The application on the client may generate new and / or modified data that is desired to be protected.

[0030] Any device in the operating environment, including clients, servers, and hosts, can take the form of software, physical machines, or virtual machines (VMs), or any combination of these devices, but no implementation requires a specific device implementation or configuration. Similarly, data protection system components such as databases, storage servers, storage volumes (LUNs), storage disks, replication services, backup servers, restore servers, backup clients, and restore clients can also take the form of software, physical machines, or virtual machines (VMs), but no implementation requires a specific component implementation. In the case of using VMs, a hypervisor or other virtual machine monitor (VMM) can be used to create and control the VMs.

[0031] As used herein, the term 'data' is intended to be broad. Thus, the term encompasses, by way of example and not limitation, data segments, data blocks, data chunks, atomic data, emails, objects of any type, files, contacts, directories, subdirectories, volumes, and any combination of one or more of the foregoing, such as may be produced by a data stream segmentation process.

[0032] Exemplary embodiments of the present invention are applicable to any system capable of storing and handling various types of objects in analog, digital or other forms. Although terms such as documents, files, blocks or objects may be used by way of example, the principles of the present disclosure are not limited to any particular form of representing and storing data or other information. Rather, such principles are equally applicable to any object capable of representing information.

[0033] Pay special attention now Figure 1 , an example of an operating environment is generally represented by 100. In some embodiments, the operating environment may include, consist of, or be a part of a data protection environment, but none of these arrangements are required. The operating environment may include an enterprise data center or a cloud data center, or both. For example, the data protection environment may support various data protection processes, including data replication, data deduplication, cloning, data backup, and data recovery. As used herein, the term backup is intended to be broadly interpreted and includes, but is not limited to, partial backup, incremental backup, full backup, clone, snapshot, continuous replication, and any other type of data copy and any combination of the foregoing. Any of the foregoing may or may not be deduplicated.

[0034] In the illustrated example, the operating environment 100 generally includes a plurality of clients 200 in communication with one or more backup servers 300. The backup servers 300 communicate with a file system synthesis server 400 and a data center 500, which in some embodiments may be a cloud data center, but this is not required. Finally, a data stream generator 600 communicates with the data center 500, the simulation server 400, and the backup servers 300. Further details on each of these components are provided in the following discussion.

[0035] The client 200 may include any number of clients from 1 to n. Figure 1 In the example of FIG. 5 , clients 202, 204, and 206 are indicated. Each of the clients 200 includes a corresponding instance of a backup agent 202a, 204a, and 206a, respectively. The backup agents 202a, 204a, and 206a cooperate with a backup application 302 on a backup server 300 to create a backup of the client 200 data for storage at a data center 500.

[0036] In addition to the backup application 302, the backup server 300 may also include a file system synthesis module 304 and a deduplication engine 306. Generally, and as discussed in more detail elsewhere herein, the file system synthesis module 304 operates to generate a simulated file system having a file structure including one or more directories and subdirectories. Likewise, the file system synthesis module 304 operates to populate the file system with data generated by the data stream generator 600. In the following Figure 2 Further details regarding an exemplary file system synthesis module 304 are set forth in the discussion of . Deduplication engine 306 deduplicates the backup (including files) generated by backup application 302 and then transmits the deduplicated data to data center 500 for storage in data center storage 501. In some implementations, file system synthesis module 304 and / or deduplication engine 306 may be hosted at respective independent servers rather than being integrated into backup server 300 or other components.

[0037] Therefore, and continue to refer to Figure 1 , some embodiments include a separate file system server 400 that communicates with the backup server 300 and transmits the set of simulated files to the backup server 300 for use by the backup application 302. In some other embodiments, the file system synthesis module 502 is included as an element of the data center 500, or the file system synthesis module 602 can be included as an element of the data flow generator 600. Likewise, in some embodiments, the data flow generator 600 can be omitted, and the data flow function is implemented by the data flow generator 504 of the data center.

[0038] If from Figure 1 As can be clearly seen from the configuration and discussion of the present invention, the functional allocation disclosed herein is presented only as an example. Generally, the disclosed functions can be allocated between the elements of the operating environment 100 in any other suitable manner, and the scope of the present invention is not limited to the disclosed configuration and arrangement.

[0039] Continue to refer Figure 1 , a brief introductory discussion of the operation of some of the disclosed elements is provided. In general, the data stream generator 600 operates to generate one or more high-speed data streams having various desired characteristics. These data streams are provided to simulation logic (such as the file system synthesis module 304), or the data streams are made accessible to the simulation logic. The file system synthesis module 304 creates a simulated file system having a file structure including files, directories, and subdirectories based on the data streams and their associated characteristics. The simulated file system is then populated with data from the data streams to create a set of files that share the data stream characteristics and can be used to test the operation and performance of the backup application 302 and / or the deduplication engine 306, etc. The set of files created by the file system synthesis module 304 can be deformed forward or backward by one or more generations by the file system synthesis module 304 using different generations of data streams and / or by using masks (examples of which are disclosed in the related applications).

[0040] Reference now Figure 2 , details of an exemplary embodiment of a simulation server 700 are provided, which may include or be composed of simulation logic, which may or may not be in the form of a file system synthesis module. In the illustrated example, the simulation server 700 may include a file structure and file engine 702, which generally operates to create a simulated file system including a specific file structure and simulated files. The specific characteristics of the file structure and the file may be specified in a property file or module 704, which is configured to receive input from a user at least in some embodiments. A fill engine 706 receives data from one or more data streams created by a data stream generator and fills the file with the data. Finally, a deformation engine 708 operates to deform the file collection created by the file structure and file engine 702 and the fill engine 706 collaboratively forward or backward by one or more generations. The deformation may be performed using data from one or more data streams and / or by using one or more masks, which indicate changes that have occurred to the data in one or more generations of data streams.

[0041] B. Example Host and Server Configuration

[0042] Now briefly refer to Figure 3, any one or more of the client 200, backup agent 202a, 204a, 206a, backup server 300, backup application 302, file system synthesis module 304, deduplication engine 306, file system synthesis server 400, data center 500, storage device 501, file system synthesis module 502, data stream generator 504, data stream generator 600, file system synthesis module 602, file structure and file engine 702, characteristic data 704, fill engine 706 and deformation engine 708 can take the form of a physical computing device, or include a physical computing device, or be implemented on a physical computing device, or be hosted by a physical computing device, an example of which is represented by 800. Similarly, in the case where any of the foregoing elements includes a virtual machine (VM) or is composed of a VM, the VM can constitute Figure 3 Virtualization of any combination of physical components disclosed in.

[0043] exist Figure 3 In the example of , the physical computing device 800 includes: a memory 802, which may include one, some or all of the following: a random access memory (RAM), a non-volatile random access memory (NVRAM) 804, a read-only memory (ROM) and a persistent memory, one or more hardware processors 806, a non-transitory storage medium 808, an I / O device 810, and a data storage device 812. One or more of the memory components 802 of the physical computing device 800 may take the form of a solid-state device (SSD) storage device. Similarly, one or more applications 814 including executable instructions are provided. Such executable instructions may take various forms, including, for example, instructions that can be executed to perform any method or part thereof disclosed herein, and / or instructions that can be executed by any storage site / at any storage site (whether it is an enterprise local, or a cloud storage site, client, data center, backup server, or network node) to perform the functions disclosed herein. Likewise, such instructions may be executable to perform part or all of any other operations and processes disclosed herein, including, but not limited to, data stream generation, file generation, file structure generation, deformation, population, deduplication, data storage and retrieval, as well as read, write, backup, clone and restore operations and / or any other data protection operations, audit operations, cloud service operations.

[0044] C. Simulate the creation of a file system

[0045] Now turn your attention to Figure 4, details about systems and processes for generating simulated file systems and files are provided, which can be used, for example, to test and evaluate the performance of applications, such as backup applications. Such file systems and files may be referred to as "simulations" herein because they are not "real world" files generated by applications operating in a production environment, but are configured to closely mimic real world files and data. As described above, the data used to populate the simulated files may be generated by a data stream generator. Before the filling process is performed, there are no actual files in the simulated file system. Instead, initially only simulated files that have not yet been filled with data exist in the simulated file system. Files and directories in the file system may be created in a variety of ways. For example, empty files and directories may be created before any data filling is performed. Alternatively, files and directories may be created and filled separately. Finally, the simulated file system may be of any type, and the scope of the present invention is not limited to any particular type of file system. Some exemplary simulated file systems include Windows NTFS, ReFS, ExFAT, Linux ext4, btrfs, zfs, and Mac.

[0046] like Figure 4 As indicated, an exemplary simulated file system 900 having a file structure 902 is disclosed. The file system 900 and associated file structure 902 may be specified by a user. In general, the file structure may be of any size and configuration and include any number and size of directories and files. Likewise, each entity or file in the file system 900 has a simulated size as specified by various input parameters, as discussed in more detail below. Thus, Figure 4 The file structure 902 shown in FIG. 9 is presented for illustration purposes only and is not intended to limit the scope of the present invention in any way.

[0047] Embodiments of the present invention are flexible in terms of file system 900 parameters that may be specified, such as by a user or by a computing entity. Those parameters include, but are not limited to: configuration of file structure 902, including structural relationships and arrangements between / more of files, directories, and / or subdirectories; file size; number of files; file types; number of directories; number of subdirectories; size / number / type of files in directories / subdirectories; specific distribution of files throughout the file structure; total size of the file system; and size and contents of directories and subdirectories. Values ​​for any of these parameters may be specified and stored for later use, or may be provided to the system in real time and inline as part of a testing and evaluation program (e.g., involving a backup application and / or a deduplication engine).

[0048] As described, the set of simulation files in the file system 900 can be used to fill the data of the data stream generated by the free data stream generator. Therefore, not only the parameters of the file system 900 can be specified, but also the parameters particularly related to the simulation files can be specified. These parameters can be referred to as file set parameters in this article because they relate to a specific file set in the file system 900. Such file set parameters include but are not limited to: the total size of the file set; the growth of the set; the average size of each entity (file) in the set; the difference of the entity size in the set; the distribution of the set entity at each folder and subfolder; and the minimum / maximum / difference at each folder level. The file set parameters are collected together with the file system parameters and used to build the simulation file system 900.

[0049] In some embodiments, the simulated file system 900 is created ad hoc based on user input. In other embodiments, the simulated file system 900 can be automatically created by picking one or more parameters from another simulated file system and / or from an actual real-world file system.

[0050] Finally, the file structure 902 can, but need not necessarily, mimic the actual or expected real-world file structure to be backed up by the backup application. Because backup applications are typically configured to combine file and file system operations, the simulated file system 900 provides a good mechanism to test the performance and operation of the backup application and / or deduplication application / engine. Therefore, as provided by embodiments of the present invention, the testing and performance results obtained using the simulated file system can provide useful insights into the performance and functionality of the backup application and deduplication application / engine. For example, one or more embodiments of the present invention can be used to determine the best and most efficient way to back up a file system. Specifically, by traversing the file structure in various ways and understanding the strengths / weaknesses of the deduplication engine, better backup mechanisms can be jointly developed.

[0051] D. Simulate the creation of a file collection

[0052] Reference now Figure 5 , provides details on an exemplary structure and process for generating a set of files using data from a data stream created by a data stream generator. Figure 5As shown, a data stream 1000 is generated that embodies various specified characteristics 1002 that are desired to be imparted to a collection of files that will be created using data from the data stream 1000. Detailed information regarding both the characteristics and generation of data streams is set forth in the related applications, and therefore, an in-depth discussion of those elements is not provided herein. However, in general, some exemplary characteristics 1002 of the data stream 1000 may include, but are not limited to, de-duplication, compression, clustering, and generality, which are specified in conjunction with the creation of the data stream 1000 and may be reflected in a collection of files that are created using data from the data stream 1000.

[0053] When data stream 1000 has been created with desired characteristics 1002, data stream 1000 is sliced ​​into a plurality of portions 1004 whose respective sizes correspond to the respective sizes of simulated files in file system 900. The collective characteristics of these portions 1004 correspond to the desired characteristics of the collection of files to be included in file system 900.

[0054] The slicing process can be performed, for example, by a file system synthesis module and / or by a data stream generator, and can be performed inline when generating a data stream or after the data stream has been generated and stored. The slicing process can be based on file characteristics specified in conjunction with the configuration of the file system, and include, for example, the total size of a file collection, collection growth, the average size of each entity (file) in the collection, and the difference in the size of entities in the collection. For illustration, the file characteristics can specify that 13 files (file A to file M) are to be created, with an average size of 20MB. Therefore, the slicing process will cut 13 fragments from the data stream, while also ensuring that the average size of the collection of 13 fragments is 20MB.

[0055] The specific portion or portions of the data stream 1000 targeted by the slicing process may vary as desired to ensure that the resulting group of 13 files in this example shares the desired characteristics 1002. Thus, the slicing process may take into account the characteristics of the data stream 1000 when deciding which data to cut from the data stream 1000 and when / where to cut. The slicing of the data stream 1000 may or may not be random. In some embodiments, slicing may target or avoid specific portions of the data stream 1000, such as the first 1 GB or the middle 10 GB, but neither approach is required. In some other embodiments, the slicing process may be performed in a uniform manner on the entire data stream 1000, which may be a virtual process.

[0056] Continue to refer Figure 4, and as previously described, the data slices obtained from the data stream 1000 are then used to populate the file system 900. Each slice corresponds to a specific file and can constitute a specific file. In some embodiments, the files in the file system 900 are created sequentially using the data obtained from the data stream. The file creation can be performed in series with the slicing process so that when slicing, the slice data is immediately used to populate the corresponding file. In other cases, one or more slices can be made and stored for later use when populating one or more files in the file collection. Once the simulated file has been filled with data from the data stream 1000, the file collection can be used for testing and evaluation of, for example, a backup application and / or a deduplication engine.

[0057] Stay tuned Figure 5 , and now also refer to Figure 6 , provides details regarding a method for creating a simulated file system and then generating a set of files from a data stream, wherein one example of such a method is generally indicated at 1100. For example, part or all of the method 1100 may be performed by a file system synthesis module.

[0058] generally, Figure 6 The method disclosed in can be used to create a collection of files that does not yet exist. Therefore, Figure 6 The method disclosed in and discussed herein may be referred to as a method for generating a basic file set. Figure 7 discussed, and through Figure 6 In contrast, an existing set of files may be forward or backward transformed into another generation of the existing set of files.

[0059] Exemplary method 1100 may begin when one or more file system parameters are received 1102, the parameters specifying various properties intended to be included in a simulated file system to be created. The simulated file system may then be created 1104 using the parameters.

[0060] Before, during, or after creating 1104 the simulated file system, target characteristics of the set of files to be stored in the simulated file system are received 1106. As described herein and in related applications, such target characteristics may include, for example, compressibility, commonality, data change rate, and clustering. Various other target characteristics of the set of files may also or alternatively be received 1106. It should be noted that processes 1102 and 1106 may be performed in parallel, or as described in the examples of FIG. Figure 6 Execute in order as shown.

[0061] Next, the data stream is sliced ​​1108 into data portions corresponding to the files to be included in the file set. Typically, files are sliced ​​1108 from the data stream one by one from a single large data stream. The data stream can be generated before or simultaneously with the slicing process 1108, and the data stream sliced ​​1108 has a target characteristic that is expected to be reflected in the file set. In at least some embodiments, the target characteristic is used as the basis for data stream generation, and / or the target characteristic is used to drive the data stream slicing process 1108 to ensure that the file set finally created reflects the target characteristic together. It should be noted that processes 1104 and 1108 can be performed on an incremental basis, i.e., at each file level.

[0062] After or during the data stream slicing process 1108, the resulting data portions of the data stream are used to populate 1110 the simulated file created 1104. In at least some embodiments, the simulated file is populated 1110 in an order corresponding to the order in which the data slices were obtained 1108 from the data stream, but this order is not required. It should be noted that the file occupies data, so the size of the file is simulated in process 1100. In addition, the set size of the simulated file system is the same as the size of the data obtained from the data stream.

[0063] After populating 1110 the simulation files with data from the data stream, the file collection may be stored and / or output 1112 to an end user. The file collection may be stored 1112 at, for example, a data center, a client, a backup server, and / or other entity in an operating environment.

[0064] Additionally or alternatively, the file collection may be transferred 1112 to an end user (such as a backup server including a backup application) for testing and evaluation 1114 purposes. For example, the backup application may use the file collection to create a backup and then transfer the backup to a deduplication engine for deduplication before storing the backup at a data center or other destination. In this manner, the operation of the backup application and / or deduplication engine may be tested using files that simulate real-world files, such as may be created by a customer at a production site and transferred to a backup server to create a backup and perform deduplication.

[0065] When the backup application first runs a file collection and backs up the file collection, the backup application scans all the files, reads the data in the files, and sends the data to the deduplication engine. Even if the cut-out order and the backup order are different from each other, the deduplication engine will treat the backup data as very close to having the same characteristics as if the original stream was written as a single large file. Therefore, when the backup application processes and sends the data, the desired deduplication / compression / versatility can still be achieved. Therefore, even if the backup application cannot directly process long data streams, it can still operate efficiently through the simulated cut-out file system.

[0066] E. Deformation of Simulated File Collection

[0067] Next go to Figure 7 , provides details about the process and arrangement of morphing a file collection from one generation forward to another. In the example shown, the file collection is morphed from generation N-1 to generation N. However, the file collection can be morphed forward multiple generations, and can be morphed backward one or more generations. Therefore, Figure 7 The examples are presented for illustrative purposes and do not limit the scope of the present invention.

[0068] In one or more of the related applications, it is noted that the data generation algorithm can regenerate a flow that depicts a known rate of change. Figure 6 The approach reapplies the same cleavage logic and recreates all files and utilizes the backup application to process these files. In this way, the reprocessing of the collection of files by the deduplication engine will generate the desired deduplication properties as well as compression / versatility and any other properties of interest. However, this approach does not reflect a typical real-life situation, as only a small fraction of the files will actually change between generations. Furthermore, this approach will require writing all files again and is therefore not particularly efficient in terms of time and utilization of processing resources.

[0069] therefore, Figure 7An exemplary method generally involves generating a data stream corresponding to the N-1 generation, and generating another stream corresponding to the N generation. The generation of the two data streams can be performed as disclosed in the related application. The two data streams are compared in sequence to identify differences between the two streams, while the method also progresses in the files being compared. When the data being compared is the same, data overwriting in the files is not performed, but when the data is different, overwriting is performed, and certain files mapped to the changed area are thereby modified. Since the streams of the N-1 generation and the N generation can be generated at a very high rate, as disclosed in the related application, and the comparison operation is also performed relatively quickly, changes to the simulated file system can take effect relatively quickly. Therefore, all files that are not mapped to the changed area remain unmodified. It should be noted that comparing data streams is a way to identify changes that need to be made in order to transform a set of files, and another way to identify changes is to use a method such as the following in combination. Figure 7 The interface discussed.

[0070] Now, when the backup application runs and performs a backup of the simulated file system, the backup application can very quickly skip all files that have not changed and read only the files that have changed. When the contents of the files that have changed are sent to the deduplication engine, the modified data has almost the same deduplication, versatility, and compressibility that is configured for the data generation algorithm that generates the data stream from which the file data is obtained. Therefore, by having two incoming data streams, and by comparing their contents, the file system can be deformed from one generation forward to another very quickly. Similarly, the same deformation logic can be applied to perform the reverse deformation process, where the data written on the mapped file is obtained from the N-1 generation instead of from the Nth generation.

[0071] It should be noted that in some embodiments, and as disclosed in the related applications, this work can be done using only the generation 0 data stream. For example, in a forward warping process where X is the value of the generation to which the file set will be forward warped from generation N, it is sufficient to have the generation X mask and the generation 0 data stream. The forward warping process can read the X mask and then use the generation 0 data stream to change any blocks of the generation N file set in the X mask that have a generation value greater than N. Finally, it should be noted that this work can be done using only the generation X mask because the mask represents the changes required after generation N, and the affected areas can be read, modified from generation N to generation X, and then rewritten. It should be noted that if the gap between N and X is very large, the last scheme should not be used because it will require an excessive amount of read / modify / writes, thereby slowing down the process.

[0072] For the reverse deformation process of reversely deforming the file set of generation X to the previous generation N of the file set, only the generation N mask and the generation X mask and the generation 0 data stream are needed. Usually, the generation N mask value and the generation X mask value are compared with each other, and the generation 0 data stream is transformed into or not transformed into the generation N state based on the result of the comparison. That is, when the generation X value of the block is the same as the generation N value of the block, the block is not restored, but when the generation X value of the block is different from the generation N value, the generation 0 block is transformed into the generation N, and then the block is written to the file to which it belongs. Later combined Fig. 9 An example of the reverse deformation process is discussed.

[0073] Now special reference Figure 7 , an exemplary embodiment 1200 involves a first set of inputs 1210 constituting features to be implemented in a corresponding data stream 1215. As noted in the related applications, the inputs 1210 may include a generated name for the data stream, (one or more) seeds used to generate the data stream, and desired compression, generality, and other features of the data stream 1215 to be generated. In this example, the data stream 1215 is an N-1 generation data stream. Each segment 1212 in the data stream 1215 represents a file boundary in the data stream. Similarly, another set of inputs 1220, which may be the same or different than the inputs 1210, constitutes features to be implemented in a corresponding Nth generation data stream 1225. Each segment 1222 in the data stream 1215 represents a file boundary in the data stream. Figure 7 As indicated, the two data streams 1215 and 1225 map to the same set of files.

[0074] Continue to refer Figure 7 , it can be seen that the comparison of the N-1 generation data stream 1215 with the N generation data stream 1225 reveals that only small portions of the two different files, file E and file I, have changed between the N-1 generation data stream 1215 and the N generation data stream 1225. That is, file E portion 1214 of the N-1 generation data stream 1215 has changed to file E portion 1217 in the N generation data stream 1225, and file I portion 1216 of the N-1 generation data stream 1215 has changed to file I portion 1219 in the N generation data stream 1225. Therefore, the N generation file set 1250N is created by overwriting portion 1214 with portion 1217 and portion 1216 with portion 1219 in the file set 1250, and no other portions of files E and I or any other files are modified, or need to be modified.

[0075] In more detail, an exemplary interface for the case where the N-1th generation data stream and the Nth generation data stream are used may take the following form:

[0076] Reset-offsets(N-1,N,other parameters)

[0077] Get-me-Next-change(offset-where-change-is,buffer)

[0078] Get-me-Next-change(offset-where-change-is,buffer)

[0079] Get-me-Next-change(offset-where-change-is,buffer)

[0080] …Done.

[0081] Using this exemplary interface, on the first call, the logic returns portion 1217 in the provided buffer and an offset corresponding to somewhere in file E. On the next call, the logic returns portion 1219 in the buffer and an offset corresponding to somewhere in file I. In either case, the logic knows exactly where the next change will occur, either by comparing the data stream or using the generation 0 data stream and the generation N-1 and generation N masks.

[0082] like Figure 7 As shown in the example of , embodiments of the present invention can be economical in terms of required time and processing resources because only the changed portion of the file is overwritten, and the entire file or file group does not need to be rewritten. In addition, because the modified data is in the form of file collection 1250, the modified data can be easily handled and processed by the backup application and deduplication engine.

[0083] You can also see that the reference Figure 7 , a comparison of the N-1 generation data stream 1215 with the N-1 generation data stream 1225 enables the file set 1250 to be reverse morphed from the N-1 generation. Specifically, the comparison of the data streams reveals that the file E portion 1217 of the N-1 generation data stream 1225 corresponds to the file E portion 1214 in the N-1 generation data stream 1215, and the file I portion 1219 of the N-1 generation data stream 1215 corresponds to the file I portion 1216 in the N-1 generation data stream 1215. Therefore, the file set 1250 can be reverse morphed from the N-1 generation to the N-1 generation by overwriting portion 1217 with portion 1214 and overwriting portion 1219 with portion 1216.

[0084] F. Aspects of Some Exemplary Methods

[0085] Reference now Figure 8, provides details regarding various aspects of an exemplary method for morphing a simulated file collection from one generation to another, wherein an example of a file collection morphing method is generally indicated at 1300. Figure 8 In the example of , the file collection of generation N is deformed forward to generation X, but as pointed out in this article, the file collection can also be deformed in the reverse direction.

[0086] Exemplary method 1300 may begin at 1302, where an Nth generation data stream and an Xth generation data stream are received. Next, the Nth generation data stream and the Xth generation data stream are compared 1304 to each other to determine 1306 whether any changes have been made to the Nth generation data stream that are reflected in the Xth generation data stream. This comparison 1304 may be performed on a block, file, or any other suitable basis. Thus, while Figure 8 The illustrative example of employs block-based comparison, but the scope of the present invention is not limited to this example.

[0087] Continue to refer Figure 8 , the determination 1306 may include a determination as to whether the Xth generation data stream block is identical to the Nth generation data stream block. If it is determined 1306 that the Xth generation data stream block is identical to the Nth generation data stream block, the method proceeds to 1308 and an overwrite of the Nth generation data stream block is not performed. On the other hand, if it is determined 1306 that the Xth generation data stream block is different from the Nth generation data stream block, the Nth generation data stream block is overwritten 1307 with the Xth generation data stream block.

[0088] Regardless of whether the block is overwritten 1307 or not overwritten 1308, method 1300 then proceeds to 1310, where it is determined whether the data stream comparison 1304 has been completed. If not, method 1300 returns to 1304, and if it has been completed, the method proceeds to 1312 to complete the transformation of the file set from generation N to generation X. It should be noted that Figure 8 , X = N + a, where a is any integer ≥ 1. Therefore, the file set can be deformed forward by any number of generations.

[0089] Continue to refer Figure 8 , it can be seen that method 1300 can be easily adapted to reverse morph a set of files of generation X to a set of files of generation N, where N=Xa, and a is any integer ≥ 1. Specifically, if it is determined at 1306 that the generation value of a block in the data stream of generation X is different from the generation value of the same block in the data stream of generation N, then in a reverse morphing scenario, the generation X version of the block is overwritten with the generation N version of the block, thereby restoring the block from generation X to generation N. In addition to this change at 1307, method 1300 is also applicable to reverse morphing a set of files from generation X to generation N.

[0090] Finally, turn your attention to Fig. 9 , provides details regarding a method and process for reverse morphing a set of files of generation X to a set of files of generation N, wherein an exemplary method is generally indicated at 1400. It should be noted that reverse morphing can be performed on a single generation or multiple generations. Fig. 9 An exemplary method and Figure 8 The exemplary methods of are similar in that both methods can be used to transform a collection of files from one generation to another. However, except Figure 8 Involves forward deformation and Fig. 9 Besides the fact that reverse deformation is involved, Figure 8 The method involves comparing data blocks from two different data streams, while Fig. 9 The methods of involve comparison of mask values. Therefore, both methods can effectively transform a collection of files from one generation to another, but they use different corresponding mechanisms to achieve it. In addition, Fig. 9 The method can be performed using only a single data stream. Typically, if a deformation requires a modified block, the generation 0 data stream is used as the basis for generating the modified block.

[0091] When receiving 1402 a 0th generation data stream, an Xth generation mask, and an Nth generation mask, the exemplary method 1400 may begin. The corresponding block generation values ​​in the Xth generation mask and the Nth generation mask are then compared 1404. If it is determined 1406 that those block generation values ​​are the same, then no change 1408 is required to the block. That is, the fact that the block generation values ​​are the same indicates that the Xth generation version and the Nth generation version of the block are the same. It should be noted that, as used herein, a "generation value" refers to the generation of a particular block of a particular data stream. Thus, for example, a 3rd generation mask entry with a generation value of 2 means that the block associated with that entry was last modified in the 2nd generation data stream.

[0092] On the other hand, if it is determined 1406 that the corresponding block generation values ​​in the Xth generation mask and the Nth generation mask are different, the corresponding block in the 0th generation data stream is transformed 1407 to the Nth generation value of the block, and the transformed block is then written to the file to which it belongs. Regardless of whether the 0th generation block is transformed and written to the file 1407 or not transformed and written to the file 1408, the method 1400 then proceeds to 1410, where a check is performed to determine whether more data is needed.

[0093] When no more data is needed, it is determined 1410 that the mask comparison and any corresponding overwrite are complete, and the process 1400 proceeds to 1412 and the process of transforming the file set from generation X to generation N is complete. On the other hand, if additional data is needed, it is determined 1410 that the mask comparison and any corresponding overwrite are not complete, and the process 1400 returns to 1404.

[0094] G. Exemplary Computing Devices and Associated Media

[0095] The embodiments disclosed herein may include the use of a special purpose or general purpose computer, including various computer hardware or software modules, as discussed in more detail below. The computer may include a processor and a computer storage medium carrying instructions, which, when executed by the processor and / or caused to be executed by the processor, performs any one or more of the methods disclosed herein.

[0096] As indicated above, embodiments within the scope of the present invention also include computer storage media, which are physical media for carrying or storing computer-executable instructions or data structures. Such computer storage media can be any available physical media that can be accessed by a general or special purpose computer.

[0097] By way of example and not limitation, such computer storage media may include hardware storage devices, such as solid state disks / devices (SSDs), RAM, ROM, EEPROM, CD-ROM, flash memory, phase change memory ("PCM"), or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other hardware storage devices, which can be used to store program code in the form of computer-executable instructions or data structures, which can be accessed and executed by general or special purpose computer systems to implement the disclosed functions of the present invention. Combinations of the above should also be included in the scope of computer storage media. Such media are also examples of non-transitory storage media, and non-transitory storage media also include cloud-based storage systems and structures, but the scope of the present invention is not limited to these examples of non-transitory storage media.

[0098] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a certain function or group of functions. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions disclosed herein are disclosed as exemplary forms of implementing the claims.

[0099] As used herein, the term "module" or "component" may refer to a software object or routine executed on a computing system. The different components, modules, engines, and services described herein may be implemented as objects or processes executed on a computing system, for example, as separate threads. Although the systems and methods described herein may be implemented in software, it is possible and contemplated to implement them in hardware or a combination of software and hardware. In the present disclosure, a "computing entity" may be any computing system as previously defined herein, or any module or combination of modules running on a computing system.

[0100] In at least some cases, a hardware processor is provided that is operable to execute executable instructions for performing a method or process, such as the methods and processes disclosed herein. The hardware processor may or may not include elements of other hardware, such as the computing devices and systems disclosed herein.

[0101] In terms of computing environment, embodiments of the present invention may be performed in a client-server environment, whether a network environment or a local environment, or in any other suitable environment. Suitable operating environments for at least some embodiments of the present invention include cloud computing environments, wherein one or more of the client, server, or other machines may reside in and operate in a cloud environment.

[0102] The present invention may be embodied in other specific forms without departing from its spirit or essential features. The described embodiments are to be considered in all respects as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the preceding description. All changes within the meaning and scope of the equivalents of the claims will be covered within the scope of the claims.

Claims

1. A non-transitory storage medium having instructions stored therein, the instructions being executable by one or more hardware processors to perform operations including: Receives a set of parameters for the simulated file system; The simulated file system is created based on the parameters, wherein: The simulated file system includes a plurality of simulated files that simulate real-world files and data; receiving a set of target characteristics comprising the set of simulation files; Slicing the data stream into a set of data slices based on the target characteristic, wherein each data slice is a size of a corresponding one of the plurality of simulation files; populating the plurality of simulation files with corresponding data slices from the set of data slices, wherein each of the plurality of simulation files contains a single data slice from the set of data slices; and Use the simulated files in the collection to test and evaluate backup applications.

2. The non-transitory storage medium of claim 1, wherein the set of target characteristics includes one or more of deduplication, compressibility, commonality, and clustering.

3. The non-transitory storage medium of claim 1, wherein the target characteristic represents a characteristic present in the data stream.

4. The non-transitory storage medium of claim 1, wherein simulation files in the collection share the set of target properties in common.

5. The non-transitory storage medium of claim 1, wherein the parameters include a configuration of a file structure of the simulated file system.

6. The non-transitory storage medium of claim 1 , further comprising receiving parameters of the set, wherein the parameters include: the total size of the collection; the growth of the collection; and the average size of each file in said collection.

7. The non-transitory storage medium of claim 1, wherein the receiving of the parameters of the simulated file system is performed in parallel with the receiving of the set of target characteristics of the collection.

8. The non-transitory storage medium of claim 1, wherein both the creation of the simulated file system and the slicing of the data stream are performed on a file level basis.

9. The non-transitory storage medium of claim 1, wherein a set size of the set is the same as a set size of the data slices obtained from the data stream.

10. A system comprising: one or more hardware processors; and The non-transitory storage medium according to any one of claims 1 to 9.

11. A method comprising the operation of any one of claims 1-9.

Images