Contiguous data storage in systems with limited storage capabilities

Abstract

Contiguous data storage in systems with limited storage capacity A technique for automatically regenerating data deleted from storage is disclosed. A request to access data is received at a first storage location. Based on a catalog and using a computer processor, it is determined that the data has been deleted from the first storage location. In response, using the computer processor, the data is automatically regenerated at the first storage location. The regeneration includes identifying a second storage location for the data based on the catalog, retrieving the data from the second storage location, and storing the data at the first storage location. The catalog is updated to reflect the regeneration of the data at the first storage location.

Description

[Technical Field]

[0001] The present invention relates to electronic data storage, and more particularly to the storage of contiguous data in systems with limited storage capabilities. [Background technology]

[0002] It is becoming increasingly common for researchers to analyze extremely large public databases (such as satellite imagery) to assist with various high-stakes tasks. In some environments, public data is combined with other information to derive additional data. However, these databases are so large that they commonly run out of storage space for the data during analysis. For example, a geographic or environmental researcher may analyze satellite imagery from across an entire country. Attempting to store information derived from satellite imagery, such as data on crop health or the status of various roads and water sources, can result in large datasets that cause researchers to run out of disk space. In existing systems, when such a problem occurs, an error occurs or a decision must be made as to which data to delete from the storage system before new data can be acquired and processed, requiring coordination between users and those responsible for the data they do not want deleted. This is time-consuming and difficult to accomplish because the data owner may not be currently nearby, and users wishing to reuse the storage space do not have access to information on how to recreate the derived information. Summary of the Invention

[0003] Embodiments include a computer-implemented method. The method includes receiving a request to access data at a first storage location. The method further includes determining, based on a catalog and using a computer processor, that the data has been deleted from the first storage location, and, in response, using the computer processor, automatically regenerating the data at the first storage location. The regeneration includes identifying, based on the catalog, a second storage location for the data, retrieving the data from the second storage location, and storing the data at the first storage location. The method further includes updating the catalog to reflect the regeneration of the data at the first storage location.

[0004] An embodiment further includes a system. The system includes a processor and a memory containing a program that, when executed by the processor, performs operations. The operations further include determining, based on the catalog and using a computer processor, that the data has been deleted from the first storage location, and, in response, using the computer processor, automatically regenerating the data at the first storage location. The regeneration includes identifying, based on the catalog, a second storage location for the data, retrieving the data from the second storage location, and storing the data at the first storage location. The operations further include updating the catalog to reflect the regeneration of the data at the first storage location.

[0005] Embodiments further include a computer program product for automatically regenerating data from storage, the computer program product being a computer-readable recording medium having computer-readable program code embodied therein, the computer-readable program code being executable by one or more computer processors to perform operations. The operations include receiving a request to access data at a first storage location. The operations include determining, based on a catalog and using a computer processor, that the data has been deleted from the first storage location, and, in response, using the computer processor, automatically regenerating the data at the first storage location, the regeneration including identifying, based on the catalog, a second storage location for the data, retrieving the data from the second storage location, and storing the data at the first storage location. The operations further include updating the catalog to reflect the regeneration of the data at the first storage location.

[0006] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates the storage of data in a system with limited storage capabilities, according to one embodiment. [Figure 2] FIG. 2 illustrates a storage service for storing data contiguously in a system with limited storage capabilities, according to one embodiment. [Figure 3] FIG. 3 illustrates software services operated by a server to store continuous data in a system with limited storage capabilities, according to one embodiment. [Figure 4]FIG. 4 is a flow chart for storing data contiguously in a system with limited storage capabilities, according to one embodiment. [Figure 5] FIG. 5 is a flow chart for regenerating data as part of continuously storing data in a system with limited storage capacity according to one embodiment. [Figure 6] FIG. 6 is a flowchart for freeing up disk space as part of storing data contiguously in a system with limited storage capabilities, according to one embodiment. [Figure 7] FIG. 7 is a flowchart for regenerating requested data as part of continuously storing data in a system with limited storage space, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] One or more embodiments disclosed herein allow a user to seamlessly store remotely accessible data at a local location without worrying about storage capacity. In embodiments, remote data sources from which a user retrieves data are recorded in a catalog system. Status and information related to the data retrieval (e.g., download time, file size, resources used) are also stored in the catalog. When faced with storage capacity limitations, the software service automatically determines which data can be most easily retrieved and regenerated. The software service then frees up storage space by deleting data that is more easily retrieved and regenerated. If the user later requests that data, it can be seamlessly retrieved and regenerated.

[0009] Additionally, in one or more embodiments, the software service may track the scripts used to process downloaded data and derive new data. Information identifying these scripts, along with the parameters provided to the scripts, may be stored and used as a factor when determining which data to delete to free up storage space. As part of regenerating the data, the service may use these scripts and any input parameters to process the downloaded data and regenerate the derived data.

[0010] FIG. 1 illustrates data storage in a system with limited storage capabilities, according to one embodiment. Computing environment 100 includes one or more users 102. These users 102 access a data retrieval service 122 to retrieve data from data sources 110. In an embodiment, data sources 110 are publicly accessible. For example, users 102 may be researchers who use publicly available geographic data. Data sources 110 may be public data repositories or websites. In another example, data sources 110 may be repositories accessible using the Internet or another wide area network. Users 102 communicate requests for data from data sources 110 using user devices (e.g., laptops, desktops, smartphones, tablets, etc.). The requests may originate from any suitable software on the user devices, such as a web browser, a search application, etc.

[0011] The data retrieval service 122 retrieves data from the data sources 110. The data retrieval service 122 then stores the data in storage 150 (e.g., an electronic database, a cloud storage location, or a file system) and uses catalog 140 to store details about the origin and retrieval of the data. In an embodiment, the information stored in catalog 140 serves as metadata identifying the origin and retrieval details of the data. For example, catalog 140 may be a table in an electronic database (e.g., a relational database, a graph database, or any other suitable database). In an embodiment, storage 150 is located on a computing device used by user 102. Alternatively, storage 150 may be accessible to user 102 using a local area network (LAN), such as an organizational network. As another alternative, storage 150 may be accessible to user 102 from a dedicated remote location (e.g., a remote electronic database, a public cloud, or a private cloud) or other suitable location.

[0012] The data retrieval history 130 may be stored in the catalog 140 and may include data regarding the origin (e.g., source) of the received data and the status of the retrieved data. For example, the data retrieval history 130 may include the information in Table 1 below.

[0013] [Table 1]

[0014] In this example, the "Data Source" row stores the publicly available address for the data. For example, this row can store the uniform resource locator (URL) and mirror URL for the data source. The "Data Size" row stores the size of the downloaded data (e.g., in bytes). The "Download Time" row stores the time used to download the data (e.g., in seconds).

[0015] The "Time for Processing" row stores the time to process the data after download. In an embodiment, user 102 can use data processing service 124 to process the downloaded data using a script (e.g., perform transformations on the data or filter the data). The original data, the processed data, or both can be stored in storage 150.

[0016] Additionally, in embodiments, data processing service 124 stores identifying information (e.g., file paths, database paths, network addresses, etc.) for any scripts used to process the downloaded data (e.g., transform the data or filter the data). For example, data processing service 124 may store information identifying these scripts or their locations in storage 150, in catalog 240, or any suitable location. In embodiments, data processing service 124 also stores input parameters provided to the scripts (e.g., in catalog 140 or another suitable location).

[0017] If the catalog is stored in a database, the "Time to Process" row can store the time required to perform this process. Additionally, the "CPU Resources Used" row can store statistics about CPU usage during processing (e.g., the number of CPUs used and the percentage of available processing power used), and the "Memory Used" row can store statistics about memory usage during processing (e.g., the percentage of available memory used). The "File System Path" row can store the path to the downloaded data (and processed data, if applicable). These are merely examples; any suitable data can be maintained in the data retrieval history 130.

[0018] In an embodiment, user 102 accesses data retrieval service 122 and data processing service 124 using a suitable network connection. For example, as discussed in more detail below with respect to FIG. 3 , in an embodiment, data retrieval service 122 and data processing service 124 are software services running on one or more computer servers (e.g., a public or private cloud computing system). In this embodiment, user 102 may access data retrieval service 122 and data processing service 124 using a LAN, a wide area network (WAN), or any other suitable network. For example, data retrieval service 122 and data processing service 124 may be accessible via the Internet, and user 102 may access these services using a suitable web services application interface (API).

[0019] FIG. 2 illustrates a software service for continuously storing data in a system with limited storage capabilities, according to one embodiment. Like the computing environment 100 illustrated in FIG. 1, the computing environment 200 includes one or more users 102. These users 102 access a data retrieval system 122 to retrieve data from data sources 110. For example, just as in FIG. 1, the users 102 may be researchers who use publicly available geographic data. The data sources 110 may be public data repositories or websites. For example, the data sources 110 may be repositories accessible using the Internet or another wide area network.

[0020] The data retrieval service 122 retrieves data from the data sources 110. The data retrieval service 122 then stores the data in storage 150 (e.g., storage 150 shown in FIG. 1) and stores details about the origin and retrieval of the data using a catalog 240. For example, like the catalog 140 shown in FIG. 1, the catalog 240 can be a table in an electronic database.

[0021] Computing environment 200 includes two additional services: a file regeneration service 226 and a file deletion service 228. In an embodiment, file deletion service 228 facilitates the deletion of data from storage 150, allowing for the contiguous storage of data. This is discussed in more detail below with respect to FIG. 6.

[0022] In an embodiment, file regeneration service 226 facilitates the regeneration of files that have been deleted by file deletion service 228. Additionally, file regeneration service 226 facilitates the use of any scripts by data processing service 124 to process downloaded data (e.g., scripts identified by data processing service 124 or another suitable service). This makes file deletion by the file deletion service transparent to the user. That is, user 102 can operate within computing environment 200 without having any knowledge of what data is actually stored in storage 150 and what data has been deleted to conserve storage space. This is discussed in more detail below with respect to FIGS. 4-7.

[0023] In an embodiment, the file regeneration service 226 uses a catalog 240 to facilitate the regeneration of files. For example, the catalog 240 may store a data retrieval history 230. The data retrieval history 230, like the data retrieval history 130 shown in FIG. 1, may include the origin (e.g., source) of the retrieved data and statistics related to the data retrieval. The data retrieval history 230 may further include data about whether previously retrieved data is stored in storage 150. This data may be used as regeneration parameters for the regeneration service. For example, the data retrieval history 230 may include the regeneration parameters in Table 2 below.

[0024] [Table 2]

[0025] As shown, Table 2 contains the same rows as Table 1 above, and also contains a "File Available?" row. In an embodiment, this row indicates whether the associated data is stored in storage 150 (e.g., whether the data has been deleted by file deletion service 228).

[0026] Additionally, Table 2 includes a row for "Processing Script(s)." In an embodiment, this row is used to store the location of the processing script(s) used by the data processing service 124 to process the data after it has been retrieved. For example, after processing the retrieved data, the data processing service 124 stores in the data retrieval history 230 the path to the script(s) used for processing (including the machine on which those scripts are stored). This path may be a file path, a database path, a network address, a URL, or any other suitable path that identifies the location of the processing script. Additionally, in an embodiment, this row (or additional rows) stores the input parameters used to operate the script. The data processing service 124 may provide these input parameters to the script when processing the retrieved data.

[0027] 3 illustrates software services operating on a server 300 for continuously storing data in a system with limited storage capabilities, according to one embodiment. Server 300 includes a processor 302, a memory 310, and a network component 320. Processor 302 generally retrieves and executes programming instructions stored in memory 310. Processor 302 includes representative examples of a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, graphics processing units (GPUs), etc. Additionally, server 300 can be a single server computer or can represent a collection of server computers.

[0028] Network component 320 includes components necessary for server 300 to interface with wireless or wired communication networks (e.g., to communicate with users 102 and data sources 110 described in FIGS. 1 and 2). For example, network component 320 may include WiFi or cellular network interface components and associated software, or wired network interface components (e.g., Ethernet components, fiber optic components, etc.).

[0029] Although memory 310 is shown as a single entity, memory 310 may include one or more memory devices having memory blocks associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of memory, such as volatile, non-volatile, or both. Memory 310 generally includes program code for performing various functions associated with the use of server 300. While the program code is generally described as various "applications" or "modules" within memory, alternative implementations may have different functionality, or combinations of functionality, or both.

[0030] Memory 310 includes a data retrieval service 122. In an embodiment, this is the data retrieval service 122 shown in Figures 1-2. As discussed in those figures, data retrieval service 122 facilitates retrieval and storage of data (e.g., from data sources 110 such as those shown in Figures 1 and 2). Memory 310 may further include a data processing service 124. In an embodiment, this is the data processing service 124 shown in Figures 1-2. As discussed in those figures, data processing service 124 processes data downloaded from data sources (e.g., by performing transformations on the data or filtering the data).

[0031] Memory 310 further includes a file removal service 228. In an embodiment, this is the file deletion service 228 shown in Figure 2. As discussed in Figure 2, file removal service 228 facilitates the deletion of previously downloaded files to conserve space while facilitating contiguous storage of data.

[0032] Memory 310 further includes a file regeneration service 226. In an embodiment, this is the regeneration service 226 shown in Figure 2. As discussed in Figure 2, file regeneration service 226 facilitates the regeneration of files that have been downloaded and subsequently deleted (e.g., using file deletion service 228) to conserve storage space.

[0033] 3, services 122, 124, 226, and 228 are implemented within server 300. This is just one example. Alternatively, services 122, 124, 226, and 228 can run on separate servers, or any combination of services can run on any number of servers. Furthermore, a particular service can be spread across multiple servers (e.g., load management).

[0034] 4 is a flowchart 400 for continuously storing data in a system with limited storage capabilities according to one embodiment. In block 402, a user (e.g., one of users 102 shown in FIGS. 1-2) requests data (e.g., from storage 150 or data source 110 shown in FIGS. 1-2). In an embodiment, the user sends the data request to a data retrieval service (e.g., data retrieval service 122 shown in FIGS. 1-3).

[0035] In block 404, the data retrieval service checks whether the requested data has metadata listed in a catalog (e.g., catalog 240 shown in FIG. 2). Data listed in the catalog may indicate, for example, that the data retrieval service has previously downloaded the requested data (e.g., the user or another user previously requested the data). For example, the data retrieval service may use the "File Available?" line shown in Table 2 to determine whether the data is available in storage 150. If block 404 answers affirmatively, flow proceeds to block 406, where the data retrieval service checks whether the requested data is available locally. For example, using the embodiment of Table 2, the catalog may include a "File Available?" line in the data retrieval history (e.g., data retrieval history 230 shown in FIG. 2).

[0036] If this block returns true, the requested data is available locally. Flow continues to block 408, where the data retrieval service returns the requested data. Alternatively, the data retrieval service can return a local file path for the requested data. In embodiments, this local file path can also be stored as part of the data retrieval history in the catalog.

[0037] Returning to block 406, if the requested data is not available locally, flow proceeds to block 410. In block 410, a file regeneration service (e.g., the file regeneration service shown in FIGS. 2-3) regenerates the requested data. In an embodiment, the file regeneration service downloads the requested data from the associated data source and performs any additional processing on the data. Additionally, the file regeneration service may use a data retrieval service and a data processing service (e.g., the data processing service 124 shown in FIGS. 1-3) to perform these operations. This is further discussed with respect to FIG. 5.

[0038] In embodiments, the file regeneration service can track data regeneration. For example, a monitoring service can provide a system administrator with identification of the regenerated data. This can be a real-time monitoring system, a log, or any other suitable system. The system administrator can use the monitoring system to identify the repeatedly regenerated data, allowing the system administrator to resolve any issues (e.g., change file deletion parameters to avoid synchronization). The monitoring system can further include CPU usage, memory usage, bandwidth usage, etc.

[0039] In this embodiment, the file regeneration system also provides an alert to a system administrator when data is regenerated. This allows the system administrator to determine when storage (e.g., storage 150 illustrated in FIGS. 1-2) does not have enough space. For example, while one or more of the techniques illustrated herein allow data to be stored continuously without error, the repeated deletion and regeneration of data can use system resources (e.g., network bandwidth for retrieving data and CPU resources for processing the data). The system administrator can use the alert to identify the data that is being repeatedly deleted and regenerated and determine whether additional space can be added (or allocated) to avoid the deletion and regeneration.

[0040] Flow then proceeds to block 408, where the file regeneration service, as described above, returns the requested data or an identifier for the requested data (e.g., a file path).

[0041] Returning to block 404, if the data does not contain associated metadata in the catalog, flow proceeds to block 412. This means, for example, that the data retrieval service has not previously downloaded the requested data. In block 412, a file deletion service (e.g., file deletion service 228 shown in FIGS. 2-3) frees up disk space (if necessary). This is discussed further below with respect to FIG. 6. In embodiments, this allows for disk space errors to be avoided. Retrieving and processing the requested data without freeing up disk space would result in using more disk space than is available, resulting in errors. The file deletion service allows for these errors to be avoided by freeing up disk space in block 412 before retrieving and processing the data.

[0042] In block 414, the data retrieval service generates the requested data. In an embodiment, the data retrieval service downloads the requested data, and the data processing service performs any additional processing on the data, which is discussed further below with respect to FIG. 7.

[0043] In block 416, the data retrieval service, or another suitable service (e.g., a file regeneration service), stores the data (e.g., in storage 150 shown in FIGS. 1-2). As described above in connection with FIG. 1, in an embodiment, the data is stored locally on the client device that requested the data (e.g., on the user's local hard drive). Alternatively, or additionally, the data is stored in a system accessible to the user (e.g., over a communications network). For example, the user may be a department of a research organization that maintains a system (e.g., a network file system, database, or cloud storage system) for use in an ongoing project. The storage location may be remote from the user and accessible to the user over a communications network.

[0044] At block 418, the data retrieval service updates the catalog. As described above in connection with FIG. 2, in an embodiment, the data retrieval service maintains a catalog (e.g., catalog 240) along with a data retrieval history (e.g., data retrieval history 230). This data retrieval history includes various data about the retrieved data. Table 2, discussed above, is one example of suitable data. At block 418, the data retrieval service updates the data retrieval history in the catalog (e.g., downloaded size, downloaded speed, data processing statistics, etc.). Flow then proceeds to block 408 and returns the data as described above.

[0045] FIG. 5 is a flowchart for data regeneration as part of continuously storing data in a system with limited storage capabilities, according to one embodiment. In an embodiment, FIG. 5 corresponds to block 410 shown in FIG. 4. In block 502, a file regeneration service (e.g., file regeneration service 226 shown in FIGS. 2-3) identifies a data source for the requested data from a catalog (e.g., catalog 140 shown in FIG. 2). In an embodiment, the file regeneration service reads the catalog entry associated with the requested data (e.g., a database table) and identifies data source information stored in the catalog (e.g., a URL or a mirror URL). For example, the file regeneration service can use the "Data Source" row shown in Table 2.

[0046] In block 504, a file re-creation service (or a file deletion service, or another suitable service) frees up the necessary disk space. In an embodiment, block 504 corresponds to block 412 shown in Figure 4, which is discussed further below with respect to Figure 6.

[0047] At block 506, the file regeneration service generates the data. In an embodiment, block 506 corresponds to block 414 shown in Figure 4. For example, at block 506, the file regeneration service may download the requested data and perform any additional processing on the data, which is discussed further below with respect to Figure 7.

[0048] 6 is a flowchart for freeing up disk space as part of contiguous data storage in a system with limited storage capabilities according to one embodiment, which corresponds to blocks 412 (shown in FIG. 4) and 504 (shown in FIG. 5).

[0049] In block 602, a file regeneration service (e.g., file regeneration service 226 shown in FIGS. 2-3), or another suitable service, determines whether disk space is available. In an embodiment, the service identifies the amount of free space available in storage (e.g., storage 150 shown in FIGS. 1-2) and the amount of space required for the retrieved data. If the data has previously been retrieved and has an associated catalog entry, the catalog is used to determine the amount of storage space required. Alternatively, the file regeneration service can determine the amount of storage space required from a data source (e.g., data source 110 shown in FIGS. 1-2).

[0050] If sufficient disk space is available, the flow ends. If sufficient disk space is not available, the flow proceeds to block 604. In block 604, a file deletion service (e.g., file deletion service 228 shown in FIGS. 2-3) identifies data to delete (e.g., one or more files). In an embodiment, the file deletion service identifies the data to delete using a catalog. The file deletion service may identify data that can be most easily regenerated based on regeneration parameters stored in the catalog (e.g., download size, download time, processing time, etc.).

[0051] For example, a file deletion service could use the "Data Size" row in Table 2 to identify the smallest amount of data to download. A file deletion service could use the "Time to Download" row in Table 2 to identify the data that will download most quickly. A file deletion service could use the "Time to Process" row in Table 2 to identify the data that will be processed most quickly. A file deletion service could use the "CPU Resource Usage" row in Table 2 to identify the data that will use the least CPU resources to process. A file deletion service could use the "Memory Usage" row in Table 2 to identify the data that will use the least memory resources to process. These are merely examples. A file deletion service could use any (or all) of these rows, along with other data, to identify data to delete.

[0052] Additionally, the file deletion service can identify infrequently used data, data that is about to become outdated, etc. For example, the catalog can track when the data was last accessed and how frequently the data is accessed (e.g., as a row in Table 2). These are merely examples, and any suitable parameters can be used.

[0053] In embodiments, the file deletion service includes default parameters for identifying data to be deleted. A user or administrator can configure these parameters (e.g., using a suitable user interface). Alternatively, the file deletion service may include parameters that cannot be changed by the user.

[0054] As another example, a file deletion service can consider costs associated with cloud storage when identifying data to delete. Many cloud systems charge for processing power, network bandwidth, storage, etc. The file deletion service can use data contained in the catalog to estimate the costs associated with recreating the deleted data and factor that into its decision. Additionally, cloud parameters (e.g., a flag indicating cloud storage, processing costs, bandwidth costs, storage costs, etc.) can be stored within the catalog itself, or another suitable location, and used by the file deletion service to generate cost estimates.

[0055] Additionally, the file deletion service may avoid deleting data that cannot be reproduced. For example, the data may not have a corresponding entry in the catalog, or the catalog entry may be incomplete. The file deletion service may avoid deleting this data. As another example, data may be marked as originating from a source that will not be available in the catalog in the future. For example, the catalog may include additional fields (e.g., rows or columns) for this information. The file deletion service may also avoid deleting this data.

[0056] The file deletion service can also verify that the data is still available from the data source before deleting the data. For example, the file deletion service can attempt to initiate a search for the requested data (e.g., data source 110 shown in FIGS. 1-2) using a location contained in the catalog. If this is unsuccessful (e.g., the file deletion service receives an error), the file deletion service can avoid deleting the data.

[0057] As another alternative, the catalog may include a field that identifies whether data is available from the data source 110. Instead of (or in addition to) checking availability before deletion, the file deletion service may operate and update this field periodically in the background to identify data that is no longer available. For example, the file deletion service may operate to check the current availability of data identified in the catalog when system usage is low or has registered time.

[0058] In block 606, the file deletion service deletes the identified data from storage (e.g., storage 150 shown in FIGS. 1-2). For example, the file deletion service may use the "File System Path" row shown in Table 2 to identify the storage location for the identified data. The file deletion service may then delete the data.

[0059] In block 608, the file deletion service updates the catalog. For example, the catalog may include an entry that identifies whether the data is available in storage (e.g., the "File Available?" row shown in Table 2 above). The file deletion service toggles this value from true (e.g., available) to false (e.g., unavailable). Flow then returns to block 602.

[0060] 7 illustrates a flowchart for generating requested data as part of a contiguous data store in a system with limited storage capabilities, according to one embodiment. In an embodiment, FIG. 7 corresponds to blocks 414 (shown in FIG. 4) and 506 (shown in FIG. 5). In block 702, a data retrieval service (e.g., data retrieval service 122 shown in FIGS. 2-3) retrieves the requested data from a data source (e.g., a data source shown in FIGS. 1-2).

[0061] In block 704, a data processing service (e.g., data processing service 124 shown in FIGS. 2-3) identifies any data processing scripts associated with the requested data. For example, a user may have identified a filter, transform, or other processing script to operate on the data after it has been retrieved and may have provided input parameters for the script. In an embodiment, the data processing script is stored as a series of steps (e.g., a human-readable programming language script or a machine-readable compiled script). The script may be stored in a file, a database table, or any other suitable format. Alternatively, the script may be available at a network location and can be invoked using the data processing service (e.g., using a network API). Any input parameters may also be stored within the script itself, in a database table, or in any other suitable location. For example, the "Processing Script(s) Shown in Table 2" row may identify a path (e.g., a file path, database path, or network path) for the script along with any input parameters.

[0062] In an embodiment, the data was previously generated and is currently being regenerated (e.g., as described above with respect to FIG. 5). The file regeneration service uses information in the catalog to identify the script (e.g., a file path or a database path, as described above with respect to FIG. 2) and, if necessary, retrieves the script and any input parameters used by the script. Alternatively, the file regeneration service, or another suitable service (e.g., a data processing service), can query the user to identify an existing script or to enter the desired script steps. In an embodiment, the file regeneration service (or another suitable service) can store the script steps and storage paths in the catalog so the script can be used even if the data is later regenerated.

[0063] In block 706, the data processing service runs the script to process the retrieved data. For example, if a script was retrieved from storage, the data processing service can run the script with input parameters (e.g., a Python script or any other suitable script). As another example, if the script is available at a network location, the data processing service can call the network location, pass it the input parameters, and receive data output (or receive an identifier for the location of the data output). The data processing service (or another suitable service) can store the processed data along with (or instead of) the received data.

[0064] The description of various embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limiting to the disclosed embodiments. Many modifications or variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements of the present embodiments beyond those found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0065] In the foregoing, reference has been made to the embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to the specifically disclosed embodiments. Instead, any combination of any of the features and elements described above, whether associated with different embodiments, is contemplated for implementing and practicing the contemplated embodiments. Moreover, while the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Accordingly, the present aspects, features, embodiments, and advantages disclosed above are merely exemplary and should not be considered elements or limitations of the appended claims except as expressly recited in the claim(s). Similarly, reference to "the present invention" should not be construed as generalizing to any inventive subject matter disclosed herein, and should not be considered elements or limitations of the appended claims except as expressly recited in the claim(s).

[0066] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (such as firmware, embedded software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system."

[0067] The present invention may be a system, method, or computer program product, or any combination thereof, including a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform the features of the present disclosure.

[0068] A computer-readable storage medium may be any tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electro-magnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of computer-readable storage media include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within grooves that record instructions, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave such as a wave guide or other communication medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal communicated through a wire.

[0069] The computer program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or can be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or combinations thereof. The network can include copper cables, fiber optics, wireless networks, routers, firewalls, switches, gateway computers, and edge servers, or combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the computing / processing device for storage.

[0070] Computer-readable program instructions for carrying out the operations of the present invention can be either source code or object code written in any combination of programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or one or more conventional procedural programming languages ​​such as Smalltalk®, object-oriented programming languages ​​such as C++, the "C" programming language, or similar programming languages. The computer-readable program instructions can execute entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection can be to an external computer (e.g., through an Internet service provider). In some embodiments, computer-readable program instructions can be executed by electrical circuitry, including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), using state information from the computer-readable program instructions to personalize the electrical circuitry to perform features of the present invention.

[0071] Aspects of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer-readable storage media and computer program products according to embodiments of the invention. It will be understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0072] The computer-readable program instructions can be provided to a general-purpose computer, a special-purpose computer, or other processor or other programmable data processing device to create a machine, and execution by the computer's processor or other programmable data processing device creates means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or a combination thereof. These computer-readable program instructions that direct a computer, programmable data processing device, or other device, or a combination thereof, to function in a particular manner can also be stored on a computer-readable recording medium, and the computer-readable recording medium having the instructions stored thereon constitutes an article of manufacture containing instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or a combination thereof.

[0073] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a computer-implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device to implement the functions / acts identified in a block or blocks of the flowcharts and block diagrams, or a combination thereof, on the computer, other programmable apparatus, or other device.

[0074] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which contain one or more executable instructions for implementing a specific logical function(s). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may actually be performed as a single step, or may be performed simultaneously, substantially simultaneously, or partially or completely overlapping in time, depending on the functionality involved, or the blocks may sometimes be performed in reverse order. Furthermore, the illustration of block diagrams and / or flowcharts and / or combinations thereof indicates that the blocks in the block diagrams and flowcharts may be implemented by special-purpose hardware-based systems that perform specific functions or operations or execute specific-purpose hardware and computer instructions.

[0075] Embodiments of the present invention can be delivered to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing can be defined as a computing capability that provides abstraction between computing resources and the underlying technical architecture (e.g., servers, storage, and networks), enabling convenient, on-demand, network access to a configurable shared pool of rapidly provisioned and open-source configurable computing resources with minimal management effort or interaction with a service provider. Thus, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even fully virtualized computing systems) in the "cloud" without regard for the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0076] Typically, cloud computing resources offer users a pay-per-use model, where users are charged only for the computing resources actually used (e.g., the amount of storage space used by the user or the number of virtualization systems instantiated by the user). In the context of the present invention, users can access applications (e.g., services 122, 124, 126, and 228 shown in FIG. 3) or associated data in the cloud. For example, data retrieval service 122, data processing service 124, file regeneration service 226, and file deletion service 228 can run on computing systems in the cloud to facilitate continuous data storage in systems with limited storage capabilities. Doing so allows users to access this information from any computing system connected to a network connected to the cloud (e.g., the Internet).

[0077] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be made without departing from the basic scope thereof, the scope of the present invention being determined by the following claims.

Claims

1. 1. A computer-implemented method comprising: Receiving a request to access data at a first storage location; determining, based on the catalog and using a computer processor, that the data has been deleted from the first storage location, and in response, automatically regenerating the data at the first storage location using the computer processor, wherein the regeneration includes: identifying a second storage location for the data based on the catalog; retrieving the data from the second storage location; storing said data in said first storage location; updating the catalog to reflect the regeneration of the data at the first storage location; and Identifying the script used to process said data; Including, Identifying the script is based on information stored in the catalog. method.

2. The first storage location is not publicly accessible, the second storage location is publicly accessible, and the catalog comprises at least one of: (i) a table in an electronic database; or (ii) metadata associated with the requested data. The method of claim 1 , comprising:

3. The data further comprises: identifying a first amount of storage space required to store the data at the first storage location and a second amount of storage space available at the first storage location; determining free storage space at the first storage location based on the first amount and the second amount; identifying files to delete from the first storage location based on the catalog; deleting the file from the first storage location; and updating the catalog to reflect the deletion of the file from the first storage location. The method of claim 1 , comprising:

4. 1. A computer-implemented method comprising: Receiving a request to access data at a first storage location; determining, based on the catalog and using a computer processor, that the data has been deleted from the first storage location, and in response, automatically regenerating the data at the first storage location using the computer processor, wherein the regeneration includes: identifying a second storage location for the data based on the catalog; retrieving the data from the second storage location; storing said data in said first storage location; updating the catalog to reflect the regeneration of the data at the first storage location; and identifying files to delete from the first storage location based on the catalog, the identifying including determining that the catalog includes a publicly accessible third storage location for the files. A method comprising:

5. Identifying the files to delete from the first storage location based on the catalog further comprises: Selecting a file to be deleted from among a plurality of files identified in the catalog based on one or more regeneration parameters associated with the file and in the catalog. Includes The method according to claim 4.

6. The regeneration parameters include one or more of: (i) the date the file was last accessed; (ii) a size associated with the file; (iii) the time to download the file; (iv) the time to process the file; (v) the amount of processing resources used to process the file; or (vi) the amount of memory resources used to process the file. The method according to claim 5.

7. Automatically regenerating the data at the first storage location further comprises: processing the retrieved data using the script; and storing the processed data at the first storage location. Includes The method of claim 1.

8. Processing the retrieved data using the script further comprises: identifying at least one input parameter associated with said processing; and Providing said at least one input parameter to said script. Includes The method of claim 7.

9. The script includes at least one of: (i) a file that includes one or more processing steps; or (ii) a software service that is accessible at a network location. The method of claim 7.

10. 1. A system comprising: A processor; A memory containing a program that, when executed by the processor, performs operations, the operations including: Receiving a request to access data at a first storage location; determining, based on the catalog and using a computer processor, that the data has been deleted from the first storage location, and in response, automatically regenerating the data at the first storage location using the computer processor, wherein the regeneration includes: identifying a second storage location for the data based on the catalog; retrieving the data from the second storage location; storing said data in said first storage location; updating the catalog to reflect the regeneration of the data at the first storage location; and Identifying the script used to process said data; Including, The identification is based on information stored in the catalog. system.

11. The first storage location is not publicly accessible, the second storage location is publicly accessible, and the catalog comprises at least one of: (i) a table in an electronic database; or (ii) metadata associated with the requested data. The system of claim 10 , comprising:

12. The data further comprises: identifying a first amount of storage space required to store the data at the first storage location and a second amount of storage space available at the first storage location; determining free storage space at the first storage location based on the first amount and the second amount; identifying files to delete from the first storage location based on the catalog; deleting the file from the first storage location; and updating the catalog to reflect the deletion of the file from the first storage location. The system of claim 10 , comprising:

13. 1. A system comprising: A processor; A memory containing a program that, when executed by the processor, performs operations, the operations including: Receiving a request to access data at a first storage location; determining, based on the catalog and using a computer processor, that the data has been deleted from the first storage location, and in response, automatically regenerating the data at the first storage location using the computer processor, wherein the regeneration includes: identifying a second storage location for the data based on the catalog; retrieving the data from the second storage location; storing said data in said first storage location; updating the catalog to reflect the regeneration of the data at the first storage location; and Identifying files to delete from the first storage location based on the catalog. Including, Identifying the files to delete from the first storage location based on the catalog further comprises: determining that the catalog includes a publicly accessible third storage location for the file; Including, the system.

14. Identifying the files to delete from the first storage location based on the catalog further comprises: Selecting a file to be deleted from among a plurality of files identified in the catalog based on one or more regeneration parameters associated with the file and in the catalog. The system of claim 13 , comprising:

15. The regeneration parameters include one or more of: (i) the date the file was last accessed; (ii) a size associated with the file; (iii) the time to download the file; (iv) the time to process the file; (v) the amount of processing resources used to process the file; or (vi) the amount of memory resources used to process the file. The system of claim 14.

16. Automatically regenerating the data at the first storage location further comprises: processing the retrieved data using the script; and storing the processed data at the first storage location. Includes 11. The system of claim 10.

17. Processing the retrieved data using the script further comprises: identifying at least one input parameter associated with said processing; and Providing said at least one input parameter to said script. Includes 17. The system of claim 16.

18. The script includes at least one of: (i) a file that includes one or more processing steps; or (ii) a software service that is accessible at a network location.

18. A system according to claim 16 or 17.

19. A computer program comprising a program code adapted to carry out the method according to any one of claims 1 to 9.

Images