Continuous storage of data in a system with limited storage capacity

Abstract

A technique for automatically recreating data removed from a storage device is disclosed. A request to access data at a first storage location is received. Based on a directory and using a computer processor, it is determined that the data has been removed from the first storage location. In response, the data is automatically recreated at the first storage location using the computer processor. The recreating includes identifying a second storage location for the data based on the directory, retrieving the data from the second storage location, and storing the data at the first storage location. The directory is updated to reflect the recreating of the data at the first storage location.

Description

Background Art

[0001] The present disclosure relates to electronic data storage, and more particularly to the continuous storage of data in systems with limited storage capacity.

[0002] It is becoming more and more common for researchers to analyze very large public data sets (such as satellite images) to assist various key tasks. In some cases, public data is combined with other information to derive additional data. However, because these data sets may be very large, the storage space used for the data is usually exhausted during the analysis. For example, geological or environmental researchers can analyze satellite images from an entire country. Trying to save information derived from satellite images, such as data about the health of crops, the status of various roads and water bodies, etc., can produce such a large data set that researchers run out of disk space. In existing systems, when similar problems occur, errors may occur or users must decide which data to delete from the storage system before new data can be ingested and processed, which may require coordination between users and result in the deletion of undesirable data. This is time-consuming and may be difficult to achieve because the data owner may no longer be nearby, and users who wish to reclaim storage space cannot obtain information about how to regenerate the exported information. Summary of the invention

[0003] Embodiments include a computer-implemented method. The method includes receiving a request to access data at a first storage location. The method also includes: determining, based on a directory and using a computer processor, that the data has been removed from the first storage location, and in response, automatically recreating the data at the first storage location using the computer processor. The recreating includes: identifying a second storage location for the data based on the directory; retrieving the data from the second storage location; and storing the data at the first storage location. The method also includes updating the directory to reflect the recreating of the data at the first storage location.

[0004] Embodiments also include a system. The system includes a processor and a memory containing a program that, when executed on the processor, performs an operation. The operation includes receiving a request to access data at a first storage location. The operation also includes: determining, based on a directory and using a computer processor, that the data has been removed from the first storage location, and in response, automatically recreating the data at the first storage location using the computer processor. The recreating includes: identifying a second storage location for the data based on the directory; retrieving the data from the second storage location; and storing the data at the first storage location. The operation also includes updating the directory to reflect the recreating of the data at the first storage location.

[0005] Embodiments also include a computer program product for automatically recreating data removed from a storage device, the computer program product including a computer-readable storage medium containing a computer-readable program code executable by one or more computer processors to perform an operation. The operation includes receiving a request to access data at a first storage location. The operation also includes: determining, based on a directory and using a computer processor, that the data has been removed from the first storage location, and in response, automatically recreating the data at the first storage location using the computer processor. The recreating includes: identifying a second storage location for the data based on the directory; retrieving the data from the second storage location; and storing the data at the first storage location. The operation also includes updating the directory to reflect the recreating of the data at the first storage location. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0007] Figure 1 The storage of data in a system with limited storage capacity according to one embodiment is shown.

[0008] Figure 2 A software service for continuous storage of data in a system with limited storage capacity is shown according to one embodiment.

[0009] Figure 3 A server running a software service for continuous storage of data in a system with limited storage capacity is shown according to one embodiment.

[0010] Figure 4 is a flow chart of continuous storage of data in a system with limited storage capacity according to one embodiment.

[0011] Figure 5 is a flow chart for regenerating data as part of continuous storage of data in a system with limited storage capacity according to one embodiment.

[0012] Figure 6 is a flow chart for freeing up disk space as part of contiguous storage of data in a system with limited storage capacity according to one embodiment.

[0013] Figure 7 is a flow chart for regenerating requested data as part of continuous storage of data in a system with limited storage capacity according to one embodiment. DETAILED DESCRIPTION

[0014] One or more embodiments disclosed herein relate to a software service that allows a user to seamlessly store remotely accessible data in a local location without worrying about storage capacity. In an embodiment, such remote data sources are recorded in a directory system, where the user has retrieved data from the remote data source. Statistics and information associated with data retrieval (e.g., download time, file size, resources used) are also stored in the directory. When faced with limited storage capacity, the software service uses this information to automatically determine which data can be most easily retrieved and recreated. The software service can then free up storage space by deleting data that is easier to retrieve and recreate. If the user requests the data later, the data can be seamlessly retrieved and recreated.

[0015] Additionally, in one or more embodiments, the software service can track scripts used to process the downloaded data and derive new data. Information identifying these scripts can also be saved along with the input parameters provided to the scripts and can be used as a factor when determining which data to delete to free up storage space. As part of the recreation of the data, the service can use these scripts and any input parameters to process the downloaded data and recreate the exported data.

[0016] Figure 1 The storage of data in a system with limited storage capacity according to one embodiment is shown. The computing environment 100 includes one or more users 102. These users 102 access a data retrieval service 122 to retrieve data from a data source 110. In an embodiment, the data source 110 is publicly accessible. For example, the user 102 can be a researcher using publicly available geographic data. The data source 110 can be a public data repository or website. As another example, the data source 110 can be a repository accessible using the Internet or other wide area network. The user 102 can use a user device (e.g., a laptop computer, a desktop computer, a smart phone, a tablet computer, etc.) to send a request for data from the data source 110. The request can come from any suitable software on the user device, including a web browser, a research application, etc.

[0017] The data retrieval service 122 retrieves data from the data source 110. The data retrieval service 122 then stores the data in a storage device 150 (e.g., an electronic database, a cloud storage location, or a file system) and uses a directory 140 to store details about the source and retrieval of the data. In an embodiment, the information stored in the directory 140 acts as metadata that identifies the details about the source and retrieval of the data. For example, the directory 140 can be a table in an electronic database (e.g., a relational database, a graph database, or any other suitable database). In an embodiment, the storage device 150 is located on a computing device used by the user 102. Alternatively, the user 102 can access the storage device 150 using a local area network (LAN) (e.g., an enterprise network). As another alternative, the user can access the storage device 150 from a dedicated remote location (e.g., a remote electronic database, a public cloud, or a private cloud) or other suitable location.

[0018] The data retrieval history 130 may be stored in the catalog 140 and may include data about the source (eg, source) of the received data and statistics about the retrieved data. For example, the data retrieval history 130 may include the information in the following table T1:

[0019] Data Source <url>, <URL: Mirror>, … < / url> Data size: <Byte> Download time: <second> Processing time: <second> CPU resources used: <# of CPUs>, <Usage %> Memory used: <% Use> File system path: <Path 1>, <Path 2>,...

[0020] Table T1

[0021] In this example, the "Data Source" row stores the address of a publicly available source of data. For example, this row can store a uniform resource locator (URL) and a mirror URL for the data source. The "Data Size" row stores the size of the downloaded data (e.g., in bytes). The "Download Time" row stores the time it took to download the data (e.g., in seconds).

[0022] The “Processing Time” row stores the time (e.g., in seconds) to process the data after downloading. In an embodiment, user 102 may use data processing service 124 to process the downloaded data using a script (e.g., perform a transformation on the data or filter the data). The raw data and the processed data or both may be stored in storage device 150.

[0023] Additionally, in an embodiment, data processing service 124 stores identification information (e.g., file paths, database paths, network addresses, etc.) of any scripts used to process the downloaded data (e.g., perform transformations on the data or filter the data). For example, data processing service 124 may store these scripts or information identifying the locations of the scripts in storage device 150, in directory 240, or in any other suitable location. In an embodiment, data processing service 124 also stores (e.g., in directory 140 or other suitable location) input parameters provided to the scripts.

[0024] When the directory is saved in the database, the "Processing Time" row can store the time required to perform the process. In addition, the "CPU Resources Used" row can store statistics on CPU usage during the process (e.g., the number of CPUs used and the percentage of available processing power used), and the "Memory Used" row can store statistics on memory usage during the process (e.g., the percentage of available memory used). The "File System Path" row can store the path of the downloaded data (and, if applicable, the processed data). These are examples only, and any suitable data can be maintained in the data retrieval history 130.

[0025] In an embodiment, user 102 may use a suitable network connection to access data retrieval service 122 and data processing service 124. For example, as described below with respect to Figure 3 As discussed in more detail, in an embodiment, the data retrieval service 122 and the data processing service 124 are software services that run on one or more computer servers (e.g., on a public or private cloud computing system). In this embodiment, the user 102 can access the data retrieval service 122 and the data processing service 124 using a LAN, a wide area network (WAN), or any other suitable network. For example, the data retrieval service 122 and the data processing service 124 can be accessed via the Internet, and the user 102 can access these services using a suitable web service application programming interface (API).

[0026] Figure 2 A software service for continuous storage of data in a system with limited storage capacity according to one embodiment is shown. Figure 1 Similar to the computing environment 100 shown in FIG. 1 , the computing environment 200 includes one or more users 102. These users 102 access the data retrieval service 122 to retrieve data from the data source 110. For example, as Figure 1 As in the example, user 102 may be a researcher using publicly available geographic data. Data source 110 may be a public data repository or website. For example, data source 110 may be a repository accessible using the Internet or other wide area network.

[0027] The data retrieval service 122 retrieves data from the data source 110. The data retrieval service 122 then stores the data in a storage device 150 (e.g., Figure 1 150) and uses a directory 240 to store details about the source and retrieval of data. Figure 1 Like directory 140 shown in , directory 240 may be a table in an electronic database.

[0028] The computing environment 200 includes two additional services: a file re-creation service 226 and a file removal service 228. In an embodiment, the file removal service 228 facilitates the removal of data from the storage device 150 to allow for continued storage of the data. This is described below with respect to Figure 6 Discuss in more detail.

[0029] In an embodiment, the file re-creation service 226 facilitates the re-creation of files that have been deleted by the file removal service 228. In addition, the file re-creation service 226 facilitates the processing of the downloaded data by the data processing service 124 using any scripts (e.g., scripts identified by the data processing service 124 or other suitable services). This makes the removal of files by the file removal service transparent to the user. That is, the user 102 can operate in the computing environment 200 without knowing what data is actually stored in the storage device 150 and what data has been removed to save storage space. This is discussed below with respect to Figure 4-Figure 7 Discuss in more detail.

[0030] In an embodiment, the file reconstruction service 226 uses the directory 240 to facilitate the reconstruction of files. For example, the directory 240 can store the data retrieval history 230. Figure 1 Like the data retrieval history 130 shown in , the data retrieval history 230 may include data about the origin (e.g., source) of the received data and statistics about the retrieved data. The data retrieval history 230 may also include data about whether the previously retrieved data is stored in the storage device 150. This data may be used as a re-creation parameter for the re-creation service. For example, the data retrieval history 230 may include the re-creation parameters in the following table T2:

[0031] Table T2

[0032] As shown, table T2 includes the same rows as table T1 described above, and also includes a "File Available?" row. In an embodiment, this row indicates whether the associated data is stored in storage device 150 (eg, whether the data has been removed by file removal service 228).

[0033] In addition, table T2 includes a "processing script" row. In an embodiment, this row is used to store the location of the processing script used by the data processing service 124 to process the data after it has been retrieved. For example, the data processing service 124 can store the path of the script (including the machine storing these scripts) used for processing in the data retrieval history 230 after processing the retrieved data. The path can be a file path, a database path, a network address or URL, or any other suitable path that identifies the location of the processing script. In addition, in an embodiment, this row (or additional row) stores the input parameters used to operate the script. The data processing service 124 can provide these input parameters to the script when processing the retrieved data.

[0034] Figure 3 A server 300 is shown running a software service for continuous storage of data in a system with limited storage capacity according to one embodiment. The server 300 includes a processor 302, a memory 310, and a network component 320. The processor 302 generally retrieves and executes programming instructions stored in the memory 310. The processor 302 is included to represent a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (GPU) with multiple execution paths, etc. In addition, the server 300 can be a single server computer, or can represent a collection of server computers (e.g., managed together).

[0035] The network component 320 includes a server 300 that interfaces with a wireless or wired communication network (e.g., Figure 1 and Figure 2 Components required for communicating with the user 102 and the data source 110 shown in FIG. 3 ). For example, the network component 320 may include a WiFi or cellular network interface component and associated software, or a wired network interface component (e.g., an Ethernet component, a fiber optic component, etc.).

[0036] Although memory 310 is shown as a single entity, memory 310 may include one or more memory devices having memory blocks associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory. Memory 310 typically includes program code for performing various functions related to the use of server 300. Program code is typically described as various functional "applications" or "modules" within memory 310, although alternative implementations may have different functions and / or combinations of functions.

[0037] Storage 310 includes data retrieval service 122. In an embodiment, this is Figure 1-Figure 2As discussed in those figures, the data retrieval service 122 facilitates (e.g., from Figure 1 and Figure 2 110) data retrieval and storage. The memory 310 also includes a data processing service 124. In an embodiment, this is Figure 1-Figure 2 . As discussed in those figures, the data processing service 124 processes the data downloaded from the data source (eg, performs a transformation on the data or filters the data).

[0038] The memory 310 also includes a file removal service 228. In an embodiment, this is Figure 2 The file removal service 228 shown in FIG. Figure 2 As shown in , the file removal service 228 facilitates the removal of previously downloaded files to save space while facilitating continuous storage of data.

[0039] The memory 310 also includes a file reconstruction service 226. In an embodiment, this is Figure 2 The file re-creation service 226 shown in FIG. Figure 2 As shown in , file recreation service 226 facilitates the recreation of files that have been downloaded and then removed to save storage space (eg, using file removal service 228).

[0040] like Figure 3 As shown in , services 122, 124, 226, and 228 are implemented in server 300. This is just an example. Alternatively, services 122, 124, 226, and 228 can each run on a separate server, or any combination of services can run on any number of servers. In addition, a particular service can be distributed across multiple servers (e.g., for load management).

[0041] Figure 4 4 is a flow chart 400 for continuous storage of data in a system with limited storage capacity according to one embodiment. At block 402, a user (e.g., Figure 1-Figure 2 ) requests data (e.g., from Figure 1-Figure 2 In an embodiment, a user sends a data request to a data retrieval service (e.g., Figure 1-Figure 3 ) as shown in the data retrieval service 122 .

[0042] At block 404, the data retrieval service checks whether the requested data has a directory (e.g., Figure 2The data listed in the directory indicates, for example, that the data retrieval service has previously downloaded the requested data (e.g., the user or another user has previously requested the data). For example, the data retrieval service can use the "File Available?" row shown in Table T2 to determine whether the data is available in storage device 150. If box 404 is answered in the affirmative, then the process proceeds to box 406, in which the data retrieval service checks whether the requested data is available locally. For example, using the example of Table 2, the directory may include a data retrieval history (e.g., Figure 2 row in the data retrieval history 230) shown in FIG.

[0043] If the box returns true, the requested data is available locally. The process proceeds to box 408, and the data retrieval service returns the requested data. Alternatively, the data retrieval service can return the local file path of the requested data. In an embodiment, the local file path can also be stored in the directory as part of the data retrieval history.

[0044] Returning to block 406, if the requested data is not available locally, the process proceeds to block 410. At block 410, the file reconstruction service (e.g., Figure 2-Figure 3 226) to regenerate the requested data. In an embodiment, the file reconstruction service downloads the requested data from the associated data source and performs any additional processing on the data. In addition, the file reconstruction service may use data retrieval services and data processing services (e.g., Figure 1-Figure 3 The data processing service 124 shown in FIG. 1 performs these actions. Figure 5 Further discussion.

[0045] In an embodiment, a file recreation service can track the recreation of data. For example, a monitoring service that identifies data that has been recreated can be provided to a system administrator. This can be a real-time monitoring system, a log, or any other suitable system. The system administrator can use the monitoring system to identify data that is repeatedly recreated to allow the system administrator to resolve any problems (e.g., change file removal parameters to avoid loss). The monitoring system can also include CPU usage, memory usage, bandwidth usage, etc.

[0046] In an embodiment, the file reconstruction system can also provide an alert to the system administrator when the data is recreated. This can allow the system administrator to determine whether the storage device (e.g., Figure 1-Figure 2150) does not include sufficient space. For example, while one or more of the techniques described herein can facilitate continuous storage of data without errors, repeated removal and recreation of data can use system resources (e.g., network bandwidth for retrieving data and CPU resources for processing data). System administrators can use alerts to identify data that is repeatedly removed and recreated, and determine whether additional space should be added (or allocated) to avoid the removal and recreation.

[0047] Flow then proceeds to block 408. At block 408, the file reconstruction service returns the requested data or an identifier of the requested data (eg, a file path), as discussed above.

[0048] Returning to block 404, if the data does not include the associated metadata in the directory, the process proceeds to block 412. This means, for example, that the data retrieval service has not previously downloaded the requested data. At block 412, the file removal service (e.g., Figure 2-Figure 3 228) to free up disk space (if necessary). This is described below with respect to Figure 6 Further discussion. In an embodiment, this can avoid disk space errors. Without freeing up disk space, retrieving and processing the requested data may end up using more disk space than is available, resulting in errors. The file removal service can avoid these errors by freeing up disk space at block 412 before retrieving and processing the data.

[0049] At block 414, the data retrieval service generates the requested data. In an embodiment, the data retrieval service downloads the requested data, and the data processing service performs any additional processing on the data. This is described below with respect to Figure 7 Further discussion.

[0050] At block 416, a data retrieval service or other suitable service (e.g., a file reconstruction service) stores the data (e.g., in Figure 1-Figure 2 As shown in the storage device 150 above. Figure 1 As discussed, in embodiments, data is stored locally on a client device that requested the data (e.g., on a user's local hard drive). Alternatively or additionally, the data is stored in a system accessible to the user (e.g., via a communication network). For example, the user may be part of a research organization that maintains a system for use with an ongoing project (e.g., a network file system, a database, or a cloud storage system). The storage location may be remote from the user, accessible by the user via a communication network.

[0051] At block 418, the data retrieval service updates the catalog. Figure 2 As discussed, in an embodiment, the data retrieval service maintains a directory (e.g., directory 240) with a data retrieval history (e.g., data retrieval history 230). The data retrieval history includes various data related to the retrieved data. Table 2 above is an example of suitable data. At block 418, the data retrieval service updates the data retrieval history in the directory (e.g., updates the download size, download speed, data processing statistics, etc.). The process then proceeds to block 408 and the data is returned as described above.

[0052] Figure 5 is a flow chart for regenerating data as part of continuous storage of data in a system with limited storage capacity according to one embodiment. In an embodiment, Figure 5 and Figure 4 410 shown in FIG. 502. At block 502, a file re-creation service (e.g., Figure 2-Figure 3 226) based on the directory (e.g., Figure 2 The file reconstruction service may use the directory 140 shown in Table T2 to identify the data source of the requested data. In an embodiment, the file reconstruction service reads the directory entry (e.g., a database table) associated with the requested data and identifies the data source information (e.g., a URL or a mirror URL) stored in the directory. For example, the file reconstruction service may use the "Data Source" row shown in Table T2.

[0053] At block 504, the file re-creation service (or file removal service or other suitable service) frees up the required disk space. In an embodiment, block 504 is associated with Figure 4 This corresponds to the box 412 shown in FIG. Figure 6 Further discussion.

[0054] At block 506, the file reconstruction service generates data. In an embodiment, block 506 is associated with Figure 4 For example, at block 506, the file reconstruction service may download the requested data and may perform any additional processing on the data. This is described below with respect to Figure 7 Further discussion.

[0055] Figure 6 is a flow chart for freeing up disk space as part of continuous storage of data in a system with limited storage capacity according to one embodiment. In an embodiment, Figure 6 Corresponds to ( Figure 4 ) box 412 and ( Figure 5 ) box 504 shown in .

[0056] At block 602, a file reconstruction service (e.g., Figure 2-Figure 3 ) or other suitable service to determine whether disk space is available. In an embodiment, the service identifies a storage device (e.g., Figure 1-Figure 2 150) and the amount of free space available in the storage device 150 shown in FIG. 15 and the amount of space required by the retrieved data. If the data has been previously retrieved and has an associated directory entry, the directory can be used to determine the amount of storage space required. Alternatively, the file reconstruction service can determine the amount of storage space required based on a data source (e.g., Figure 1-Figure 2 The data source 110 shown in FIG. 1 determines the amount of storage space required.

[0057] If there is enough disk space, the process ends. If there is not enough disk space, the process proceeds to block 604. At block 604, the file removal service (e.g., Figure 2-Figure 3 228) to identify data (e.g., one or more files) to be removed. In an embodiment, the file removal service uses the directory to identify the data to be removed. The file removal service can identify the data that can be most easily regenerated based on the reconstruction parameters (e.g., download size, download time, processing time, etc.) stored in the directory.

[0058] For example, the file removal service can use the "Data Size" row shown in Table T2 to identify the data that downloads the smallest amount. The file removal service can use the "Download Time" row shown in Table T2 to identify the data that downloads the fastest. The file removal service can use the "Processing Time" row shown in Table T2 to identify the data that processes the fastest. The file removal service can use the "CPU Resources Used" row in Table T2 to identify the data that uses the least CPU resources to process. The file removal service can use the "Memory Used" row shown in Table T2 to identify the data that uses the least memory resources to process. These are just examples. The file removal service can use any (or all) of these rows as well as other data to identify data to be removed.

[0059] In addition, the file removal service can identify infrequently used data, data that may be outdated, etc. For example, the directory can track the time when the data was last accessed and how often the data was accessed (e.g., as rows in table T2), and this can be used to identify infrequently used data. These are examples only, and any suitable parameters can be used.

[0060] In an embodiment, the file removal service includes default parameters for identifying data to be removed. A user or administrator may configure these parameters (e.g., using a suitable user interface). Alternatively, the file removal service may include parameters that cannot be changed by a user.

[0061] As another example, the file removal service may consider costs associated with cloud storage when identifying data to be removed. Many cloud systems charge for processing power, network bandwidth, storage, etc. The file removal service may use the data included in the directory to estimate the costs associated with recreating the removed data, and may include this cost as a factor in its decision making. Additionally, cloud parameters (e.g., flags indicating cloud storage, processing costs, bandwidth costs, storage costs, etc.) may themselves be stored in the directory or other suitable location and used by the file removal service to generate cost estimates.

[0062] In addition, the file removal service can avoid removing data that cannot be recreated. For example, the data may not have a corresponding entry in the directory, or the directory entry may be incomplete. The file removal service can avoid removing the data. As another example, the data can be marked in the directory as being obtained from a source that may not be available in the future. For example, the directory can include an additional field (e.g., row or column) with this information. The file removal service can also avoid removing the data.

[0063] The file removal service can also verify that the data is still available from the data source before removing the data. For example, the file removal service can attempt to use the location included in the directory to start retrieving the requested data (e.g., from Figure 1-Figure 2 If this is unsuccessful (eg, the file removal service receives an error), the file removal service may refrain from removing the data.

[0064] As another alternative, the directory may include a field that identifies whether the data is available from the data source 110. Instead of (or in addition to) checking availability before deleting, the file removal service may periodically operate in the background and update the field to identify data that is no longer available. For example, the file removal service may operate when the system has low usage or at specified times to check the current availability of data identified in the directory.

[0065] At block 606, the file removal service removes the file from the storage device (e.g., Figure 1-Figure 2 The file removal service may remove the identified data from the storage device 150 shown in Table T2. For example, the file removal service may use the "file system path" row shown in Table T2 to identify the storage location of the identified data. The file removal service may then remove the data.

[0066] At block 608, the file removal service updates the catalog. For example, the catalog may include an entry identifying whether the data is available in the storage device (e.g., the "File Available?" row shown in Table 2 above). The file removal service may switch the value from true (e.g., available) to false (e.g., not available). The process then returns to block 602.

[0067] Figure 7 is a flow chart for generating requested data as part of continuous storage of data in a system with limited storage capacity according to one embodiment. In an embodiment, Figure 7 Corresponds to ( Figure 4 ) box 414 and ( Figure 5 ) box 506. At box 702, a data retrieval service (e.g., Figure 2-Figure 3 ) retrieves data from a data source (e.g., Figure 1-Figure 2 The requested data is retrieved from the data source 110 shown in FIG.

[0068] At block 704, a data processing service (e.g., Figure 2-Figure 3 124) identifies any data processing script associated with the requested data. For example, a user may have identified a filtering, transformation or other processing script to be performed on the data after the data is retrieved, and may have provided input parameters for the script. In an embodiment, the data processing script is stored as a series of steps (e.g., a human-readable programming language script or a machine-readable compiled script). The script can be stored in a file, in a database table, or in any other suitable manner. Alternatively, the script is available at a network location and can be called using a data processing service (e.g., using a network API). Any input parameters may also be stored in the script itself, in a database table, or in any other suitable location. For example, the "processing script" row shown in Table T2 can identify the path (e.g., file path, database path, or network path) of the script and any input parameters.

[0069] In an embodiment, data has been previously generated and is now being regenerated (e.g., as described above with respect to Figure 5 The file reconstruction service uses the information in the directory (e.g., file path or database path, as discussed above). Figure 2 The file re-creation service (or other suitable service) may store the steps of the script, as well as the storage path in a directory, so that the script can be used if the data is regenerated later.

[0070] At block 706, the data processing service runs the script to process the retrieved data. For example, if the script is retrieved from storage, the data processing service may run the script with the input parameters (e.g., run a Python script or any other suitable script). As another example, if the script is available at a network location, the data processing service may call the network location, pass the input parameters to the network location, and receive the data output (or an identification of the location to receive the data output). The data processing service (or other suitable service) may store the processed data in addition to (or in place of) the retrieved data.

[0071] The description of various embodiments of the present invention has been presented for the purpose of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications or technical improvements to technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0072] In the above, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. On the contrary, any combination of the above-mentioned features and elements is contemplated to implement and practice the contemplated embodiments, whether or not related to different embodiments. In addition, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether a particular advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Therefore, the aspects, features, embodiments and advantages discussed above are merely illustrative and are not considered to be elements or limitations of the appended claims unless expressly stated in the claims. Similarly, reference to the "present invention" should not be interpreted as a generalization of any creative subject matter disclosed herein, and should not be considered to be elements or limitations of the appended claims unless expressly stated in the claims.

[0073] Aspects of the present invention may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be collectively referred to herein as a "circuit", "module" or "system".

[0074] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform various aspects of the present invention.

[0075] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding devices such as convex structures in punch cards or grooves with instructions recorded thereon, and any suitable combination of the foregoing. Computer readable storage medium as used herein should not be interpreted as transient signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses passing through fiber optic cables) or electrical signals transmitted by wires.

[0076] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0077] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming languages ​​or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can perform computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit, so as to perform various aspects of the present invention.

[0078] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0079] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes a product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0080] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0081] The flow chart and block diagram in the accompanying drawings show the architecture, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. To this end, each square block in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing a specified logical function. In some alternative implementations, the function marked in the square block may not occur in the order marked in the figure. For example, depending on the function involved, the two square blocks shown in succession can actually be performed substantially simultaneously, or these square blocks can sometimes be performed in reverse order. It should also be noted that each square block in the block diagram and / or flow chart and the combination of the square blocks in the block diagram and / or flow chart can be realized by a system based on special-purpose hardware, which performs a specified function or action based on the special-purpose hardware or performs a combination of special-purpose hardware and computer instructions.

[0082] Embodiments of the present invention may be provided to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as providing an abstracted computing capability between computing resources and their underlying technology architecture (e.g., servers, storage devices, networks), thereby enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in a "cloud" without regard to the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0083] Typically, cloud computing resources are provided to users on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualized systems instantiated by the user). Users can access any resource residing in the cloud at any time and from anywhere on the Internet. In the context of the present invention, users can access applications available in the cloud (e.g., Figure 3124, 226, and 228) or related data. For example, data retrieval service 122, data processing service 124, file reconstruction service 226, and file removal service 228 can be executed on a computing system in the cloud and facilitate continuous storage of data in a system with limited storage capacity. Doing so allows a user to access the information from any computing system attached to a network (e.g., the Internet) connected to the cloud.

[0084] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope of the present invention is determined by the claims that follow.

Claims

1. A computer-implemented method comprising: receiving a request to access data at a first storage location; determining, based on the directory and using a computer processor, that the data has been removed from a first storage location, and in response, automatically recreating, using the computer processor, the data at the first storage location, the recreating comprising: identifying a second storage location for the data based on the directory; retrieving the data from a second storage location; determining to release storage space at the first storage location based on a first amount of storage space required to store the data at the first storage location and a second amount of storage space available at the first storage location; identifying a file to be removed from the first storage location based on: determining, using the directory, that the file is available from a third, publicly accessible storage location that is different from the first storage location; removing the file from the first storage location; and storing the data at a first storage location; and The directory is updated to reflect the re-creation of the data at the first storage location.

2. The method according to claim 1, wherein: The first storage location is not publicly accessible, wherein the second storage location is publicly accessible, and wherein the directory includes at least one of: (i) a table in an electronic database, or (ii) metadata associated with the requested data.

3. The method according to claim 1 or 2, wherein: Recreating said data also includes: The directory is updated to reflect the removal of the file from the first storage location.

4. The method according to claim 1, wherein: Identifying the file to be removed from the first storage location further comprises: The file to be removed is selected from among a plurality of files identified in the directory based on one or more recreation parameters associated with the file and stored in the directory.

5. The method according to claim 4, wherein: The recreation parameters include one or more of: (i) a last access date for the file, (ii) a size associated with the file, (iii) a time to download the file, (iv) a time to process the file, (v) an amount of processing resources used to process the file, or (vi) an amount of memory resources used to process the file.

6. The method according to claim 1, wherein: Automatically recreating the data at the first storage location further comprises: identifying scripts used to process said data; using the script to process the retrieved data; and The processed data is stored at a first storage location.

7. The method according to claim 6, wherein: Processing the retrieved data using the script also includes: identifying at least one input parameter associated with the process; and The at least one input parameter is provided to the script.

8. The method according to claim 6 or 7, wherein: The script includes at least one of: (i) a file including one or more processing steps, or (ii) a software service accessible at a network location.

9. The method according to claim 6 or 7, wherein: The script is identified based on information stored in the directory.

10. A computer system comprising: processor; as well as a memory containing a program that, when executed on the processor, performs operations comprising: receiving a request to access data at a first storage location; determining, based on the directory and using the processor, that the data has been removed from the first storage location, and in response, automatically recreating, using the processor, the data at the first storage location, the recreating comprising: identifying a second storage location for the data based on the directory; retrieving the data from a second storage location; determining to release storage space at the first storage location based on a first amount of storage space required to store the data at the first storage location and a second amount of storage space available at the first storage location; identifying a file to be removed from the first storage location based on: determining, using the directory, that the file is available from a third, publicly accessible storage location that is different from the first storage location; removing the file from the first storage location; and storing the data at a first storage location; and The directory is updated to reflect the re-creation of the data at the first storage location.

11. The computer system according to claim 10, wherein: The first storage location is not publicly accessible, wherein the second storage location is publicly accessible, and wherein the directory includes at least one of: (i) a table in an electronic database, or (ii) metadata associated with the requested data.

12. The computer system according to claim 10 or 11, wherein: Recreating said data also includes: The directory is updated to reflect the removal of the file from the first storage location.

13. The computer system according to claim 10, wherein: Identifying the file to be removed from the first storage location further comprises: The file to be removed is selected from among a plurality of files identified in the directory based on one or more recreation parameters associated with the file and stored in the directory.

14. The computer system of claim 13, wherein: The recreation parameters include one or more of: (i) a last access date for the file, (ii) a size associated with the file, (iii) a time to download the file, (iv) a time to process the file, (v) an amount of processing resources used to process the file, or (vi) an amount of memory resources used to process the file.

15. The computer system according to claim 10, wherein: Automatically recreating the data at the first storage location further comprises: identifying scripts used to process said data; using the script to process the retrieved data; and The processed data is stored at a first storage location.

16. The computer system of claim 15, wherein: Processing the retrieved data using the script also includes: identifying at least one input parameter associated with the process; and The at least one input parameter is provided to the script.

17. The computer system according to claim 15 or 16, wherein: The script includes at least one of: (i) a file including one or more processing steps, or (ii) a software service accessible at a network location.

18. The computer system according to claim 15 or 16, wherein: The script is identified based on information stored in the directory.

19. A computer program product for automatically recreating data removed from a storage device, the computer program product comprising: Computer readable program code executable by one or more computer processors to perform operations comprising: receiving a request to access data at a first storage location; determining based on the directory that the data has been removed from the first storage location, and in response, automatically recreating the data at the first storage location, the recreating comprising: identifying a second storage location for the data based on the directory; retrieving the data from a second storage location; determining to release storage space at the first storage location based on a first amount of storage space required to store the data at the first storage location and a second amount of storage space available at the first storage location; identifying a file to be removed from the first storage location based on: determining, using the directory, that the file is available from a third, publicly accessible storage location that is different from the first storage location; removing the file from the first storage location; and storing the data at a first storage location; and The directory is updated to reflect the re-creation of the data at the first storage location.

20. The computer program product of claim 19, wherein: The first storage location is not publicly accessible, wherein the second storage location is publicly accessible, and wherein the directory includes at least one of: (i) a table in an electronic database, or (ii) metadata associated with the requested data.

21. A computer program product according to claim 19 or 20, wherein: Recreating said data also includes: The directory is updated to reflect the removal of the file from the first storage location.

22. The computer program product of claim 19, wherein: Identifying the file to be removed from the first storage location further comprises: The file to be removed is selected from among a plurality of files identified in the directory based on one or more recreation parameters associated with the file and stored in the directory.

23. The computer program product of claim 19, wherein: Automatically recreating the data at the first storage location further comprises: identifying scripts used to process said data; using the script to process the retrieved data; and The processed data is stored at a first storage location.

24. A computer-readable storage medium storing a computer program, the computer program comprising program code adapted to perform the method according to any one of claims 1 to 9 when the program is run on a computer.

Images