Method and system for recovery from a failure

Abstract

Embodiments of the present disclosure relate to disaster recovery of cloud resources. Embodiments of the invention relate to a method for different services registered to a cloud infrastructure can recover in case of a disaster. Also relates to a process for a resource provider can recover and rebuild its cloud resource information by scanning the actual available resources. The resource provider will synchronize resources with the central service to get updated consumer and subscription information. In case the central service fails and recovers from a backup, the central service should synchronize with the resource provider to update its resource information.

Description

[0001] RELATED APPLICATIONS

[0002] This application is a continuation-in-part of International Application No. PCT / US2016 / 065032, International Filing Date of 6 December 2016, entered into the National Stage in the United States on 11 June 2018, and assigned U.S. Application No. 15 / 982, 1 1, entitled "Disaster Recovery for Cloud Resources," the entire disclosure of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to disaster recovery for cloud resources. BACKGROUND

[0004] Cloud computing provides access to a wide range of services. The range of these services includes processing, servers, storage, networks, applications, and online services. These services allow cloud consumers to quickly configure, manage, and release resources as needed with little management effort and no interaction with the service provider. Providers advertise a set of services as a bundle (often referred to as a "plan" or "offer"). Cloud consumers subscribe to these bundles to get a system-wide entity called a subscription that provides the consumer access to all the advertised services. Once a subscription is acquired, the consumer can use these different services to create resources.

[0005] Cloud infrastructure includes a set of services registered by the service provider during initial setup. This set of services is updated over time (e.g., added and removed). These cloud infrastructure services have different types, from services needed to monitor, run, and manage the cloud infrastructure (often referred to as "system services") to services used directly by cloud consumers to create their own resources (often referred to as "resource providers"). There is a central service that manages all these registered services. This central service acts like a master service that provides centralized management of cloud infrastructure and resource access. All cloud infrastructure services register with the central service, and the central service decides the routing of cloud consumer requests to the relevant resource provider services.

[0006] There can be situations where one or more services go down due to software, hardware, or network failures. If the central service fails, the entire system cannot respond to cloud consumer requests and becomes unavailable. If one of the resource providers goes down, cloud consumers and service providers will not be able to manage and create resources on that failed cloud service. If a system service fails, cloud system functionality is affected, and the cloud system can not be usable. SUMMARY

[0007] This summary is provided to introduce some concepts of the present application in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0008] Embodiments of the present application relate to a method for different services registered with a cloud infrastructure to recover in case of a disaster. Also related is a process for a resource provider to recover and rebuild its cloud resource information by scanning the actual available resources. The resource provider synchronizes with the central service to get updated consumer and subscription information. In case the central service fails and is recovered from a backup, the central service should synchronize with the resource provider to update its resource information. BRIEF DESCRIPTION OF DRAWINGS

[0009] To further clarify the above and other advantages and features of the embodiments of the present application, a more particular description of the application will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the application and are therefore not to be considered limiting of its scope. The application will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0010] FIG. 1 is a high level block diagram of a system that provides enterprise users with access to local provisioned resources as well as remote or public resources.

[0011] FIG. 2 is a block diagram of a data center that provides cloud computing services or distributed computing services in accordance with one embodiment.

[0012] FIG. 3 shows the communication connection between the central service and the resource provider.

[0013] FIG. 4 shows the resource provider domain that is responsible for tracking and managing resource information.

[0014] FIG. 5 shows a series of steps for disaster recovery for the resource provider.

[0015] FIG. 6 shows a series of steps for disaster recovery for the central system.

[0016] FIG. 7 is a flow diagram showing an example method for recovery from a disaster or failure within a cloud computing infrastructure.

[0017] FIG. 8 is a flow diagram showing an example method for recovery from a failure within a distributed computer network. DETAILED DESCRIPTION

[0018] FIG. 1 is a high-level block diagram of a system that provides enterprise users with access to local on-premise resources as well as remote or public resources. Local enterprise terminals 101 allow users to directly access a local data center (private cloud) 102 via an on-premise network 103. Users located outside of the enterprise 100 can use remote terminals 104 to access the on-premise data center 102. Terminals 101 and 104 can be, for example, desktops, laptops, notebooks, or tablets. Other devices such as special purpose terminals, smart phones, personal digital assistants (PDAs), and the like can also be used as terminals 101 and 104.

[0019] A firewall 105 provides a network security system for the enterprise 100 and controls incoming and outgoing network traffic. External terminals 104 can connect to the enterprise on-premise network 103 via the Internet 106 or any public or private network. Firewall 105 allows terminals 104 to access the on-premise data center 102 if the terminals 104 provide appropriate credentials and authentication. Enterprise users at terminals 101 and 104 can also access a public data center (public cloud) 107 via the Internet 106.

[0020] The on-premise data center 102 and the public data center 107 can provide "cloud computing" services to the enterprise 100 and other users. Cloud computing provides virtually unlimited computing, storage, and network resources at a low cost by liberating enterprise users from managing information technology (IT) infrastructure, while allowing services to scale on demand.

[0021] FIG. 2 is a block diagram of a distributed computing network or data center 200 that provides cloud computing services or distributed computing services in accordance with one embodiment. A plurality of servers 201 are managed by a data center management controller 202. Load balancers 203 distribute requests and workloads across servers 201 to avoid situations in which a single server 201 becomes overwhelmed and to maximize the available capacity and performance of resources in the data center 200. Routers / switches 204 support data communications between servers 201 as well as between the data center 200 and external resources and users via an external network 205, which can be a local area network (LAN) in the case of an enterprise on-premise data center 102 or the Internet in the case of a public data center (107).

[0022] The servers 201 can be traditional standalone computing devices, and / or they can be configured as individual blades in a rack of many server devices. The servers 201 have input / output (I / O) connectors that manage communications with other database entities. One or more host processors on each server 201 run a host operating system (O / S) that supports multiple virtual machines (VMs). Each VM can run its own O / S, such that each VM O / S on a server is different, or the same, or both. For example, the VM O / Ss can be different versions of the same O / S (e.g., running different current and older versions of the same operating system on different VMs). Additionally or alternatively, the VM O / Ss can be provided by different manufacturers (e.g., some VMs run an operating system while other VMs run an operating system). Each VM can thus run one or more applications (Apps). Each server also includes storage (e.g., hard disk drives (HDDs)) and memory (e.g., RAM) that can be accessed and used by the host processor and the VMs.

[0023] Cloud computing is the delivery of computing capacity as a service such that the access to IT resources such as computing capacity, networking, and storage are available like water from a tap. As with any utility, you typically only pay for what you use of the cloud computing. By tapping into cloud services, users can harness the power of massive data centers without having to build, manage, or maintain costly and complex IT building blocks. With cloud computing, much of the complexity of IT is abstracted away, letting you focus on the infrastructure, data, and application development that really matter to your business.

[0024] The data center 200 provides pooled resources that customers or tenants can dynamically provision and scale applications on as needed without having to add more servers or additional networking. This allows tenants to get the computing resources they need without having to procure, provision, and manage infrastructure on a temporary basis for each application. Cloud computing data centers 200 allow tenants to dynamically expand or reduce resources to meet their current business needs. In addition, data center operators can offer tenants usage-based services so that they only pay for the resources they use when they need to use them. For example, a tenant can initially use one VM on a server 201-1 to run their application. When demand increases, the data center can activate additional VMs on the same server and / or on new servers 201-N as needed. If demand later decreases, these additional VMs can be deactivated.

[0025] The data center 200 can provide guaranteed availability, disaster recovery, and backup services. For example, the data center can designate one VM on server 201-1 as the primary location for a tenant's application, and if the first VM or server 201-1 fails, a second VM on the same or a different server as a backup or standby server can be activated. The database manager 202 automatically transfers incoming user requests from the primary VM to the backup VM without requiring tenant intervention. Although the data center 200 is shown as a single location, it should be understood that the servers 201 can be distributed to multiple locations globally to provide additional redundancy and disaster recovery capabilities.

[0026] Data center operators can offer different levels of cloud computing services to tenants. With infrastructure as a service (IaaS) offerings, the lower levels of the IT stack are delivered as services, freeing developers from the complexity of providing physical machines and configuring networks. With IaaS, tenants can easily provision virtual machines in a highly scalable, available cloud environment, develop and test solutions, and then deploy applications to production. With platform as a service (PaaS) offerings, all functionality from network connectivity to runtime is provided. PaaS makes development easy by providing additional support for application services and operating system management, including updates. With PaaS, tenants can focus on the business logic of an application and quickly move the application from concept to launch. With software as a service (SaaS) offerings, a single, complete application or suite of applications can be delivered to customers through a web browser, eliminating their need to manage the underlying components of the IT stack, including application code.

[0027] Referring again to FIG. 1 , an enterprise can use a public data center or public cloud computing service to take advantage of cost savings, reduced management requirements, or specific services offered. On the other hand, an enterprise can also use a pre-positioned data center or private cloud service to ensure data security or use proprietary applications, for example. It can be appreciated that an enterprise does not necessarily use a pre-positioned data center to take advantage of a private cloud service. Rather, a private cloud service can be provided by a data center that restricts access to the enterprise. The use of both public cloud services and private cloud services by an enterprise is often referred to as a hybrid cloud.

[0028] FIG. 3The communication connection between the central service 301 and the resource providers 302 is shown. The central service 301 acts as the principal and manages the list of resource providers 302, consumer subscriptions and their resources 303. The central service 301 tracks resource provider registration and subscription management. The central service 301 is the principal of the subscription. Each resource provider 302 registers with the central service 301 and enables a consumer to create a certain type of resource 303. The resource provider 302 creates the actual resource 303 on behalf of the consumer and maintains the metadata about the resource 303. The resource provider 302 acts as the principal of the resource 303. The resource 303 represents the actual resource (such as a website, database, storage, etc.) created by the consumer on the cloud infrastructure.

[0029] FIG. 4 The resource provider domain is shown in which the resource provider 401 is responsible for tracking and managing the resource information. The resource provider 401 tracks the consumer subscriptions. The resource provider 401 also creates and maintains the actual resources 403 on behalf of the consumer. The resource provider 401 holds information about the consumer access in the form of subscriptions 402. The resource consumption limits are also defined in the form of quotas in the subscription. The resource provider 401 also tracks the actual resources 403 and the resource metadata. The resource provider 401 tags the actual resources 403 or names the resources 403 in such a way that the metadata records of the actual resources can be recreated from this information. Thus, in the recovery process, the resource provider 401 will regenerate the metadata by scanning the existing resources 403.

[0030] FIG. 5 A series of steps for disaster recovery for a resource provider is shown. The recovery scheme can depend on the location of the actual resources and the central service. In step 501, the resource provider has failed. There are many reasons for a cloud service to fail, such as hardware, network, communication link or software failure. In step 502, the resource provider is recovered from the backup. At this point, the resource provider will need to rebuild its metadata repository. In step 503, the service provider reconfigures the resource provider to perform the recovery process and synchronize with the central service.

[0031] In step 504, the resource provider will start scanning the actual resources to build the resource metadata. The resource metadata will be created using the consumer subscriptions so that it can provide access to them to the consumers. If the metadata discovery and recovery results in any business rule violations, these metadata records will be correctly flagged so that the service provider can take manual action for recovery.

[0032] In step 505, the resource provider synchronizes with the central service to update the consumer subscription information. It will update the subscription information in the metadata repository. Any records that appear to conflict with the business rules will be flagged as unrecoverable.

[0033] FIG. 6 A series of steps for disaster recovery of the central service is shown. The recovery process depends on the underlying resource providers. In step 601, it is detected that the central service has failed. In step 602, the central system recovers from the backup. In step 603, the service provider reconfigures the central service with the latest settings to ensure that all the latest configurations are applied. The service provider re-registers all the resource providers so that they can point to the latest endpoints and verify that the central service can communicate with the resource providers.

[0034] The central system will then need to update its consumer subscription information. In step 604, the central service synchronizes with the resource providers and each resource provider provides a list of the consumer subscriptions they are aware of. Based on the available information, the central system needs to update the subscriptions or create new offers (i.e., plans and promotions) and associate the discovered subscriptions with the offers. These subscriptions and offers are marked in a way that the service provider can later review them and take manual action.

[0035] The central system needs to keep track of the resources owned by the consumers. In step 605, the central system synchronizes with each registered resource provider to update its resource metadata information. If there are any conflicts during the metadata recovery, the record will be marked for review by the service provider.

[0036] In one embodiment, one or more computer storage media embody computer-executable instructions that, when executed, perform a method for recovery from a disaster within a cloud computing infrastructure. In the method, a tenant acquires a subscription from a service provider. The tenant uses the subscription to create a resource on a resource provider. The resource provider will create the resource and tag it with specific information that helps uniquely identify the resource and enables the resource provider to determine who owns this resource. During disaster recovery, the resource provider will use this information to recreate its resource metadata records by scanning the actual resources and the information tagged from these resources. This tagging is necessary for successful disaster recovery when the resource registration of the resource provider needs to be recovered.

[0037] The resource providers can maintain a list of consumer subscriptions and resources created by the consumers. Each resource provider will expose a list of consumer subscriptions that have access to that resource provider. The resource provider will also provide a list of resources owned by each consumer subscription. The central service will use the subscription and resource information to rebuild its subscription and resource metadata information during recovery. The recovered resources and subscriptions can be marked for subsequent review.

[0038] The central service will synchronize with the resource providers to update its resource and subscription information. Once the central service recovers from the failure, the service provider can reconfigure the central service and ensure that the resource provider information is up to date. The central service looks for all registered resource providers to synchronize the tenant subscriptions and resources. The resource providers expose the resource and subscription information. The central service can tag the recovered resources in a way that enables the service provider and tenants to audit them.

[0039] There are two ways in which the system can get out of sync:

[0040] (1) The centralized service has information that the resource provider does not. The centralized service creates the missing subscription and synchronizes it, which will result in deleting (or tagging for deletion) any additional information about the resources that are missing in the resource provider.

[0041] (2) The resource provider has information about a subscription that the centralized service does not. The centralized service recreates the record about this subscription to further facilitate troubleshooting or clean deletion.

[0042] If neither the resource provider nor the centralized service has information about a subscription, its trail can be lost. There can still be information about such a subscription in the auxiliary services (such as monitoring, logging, billing, etc.), and similar synchronization needs to be performed.

[0043] The central service can need to synchronize with other services that are located "above" it, such as a billing system that approves subscription creation. The process of such synchronization is similar to the one described above. Subscriptions that are not in the billing system are suspended or tagged for audit / deletion. Subscriptions that are in the centralized service but not in the billing system are also suspended, and manual information is needed to fix them.

[0044] When critical information about a subscription is missing and it is not possible to recover it automatically (for example, when the owner or provider information is missing), manual recovery can be needed:

[0045] For subscription recovery - it can be needed to create a new subscription.

[0046] For resource recovery - the resources from the recovered subscription can need to be moved to a newly recreated subscription at the time of verification of resource ownership.

[0047] FIG. 7is a flowchart illustrating an example method for recovery from a disaster or failure within a cloud computing infrastructure. In step 701, resources are created by a resource provider on a distributed computing network. The resources each have a metadata record. In step 702, the resources are tagged by the resource provider with specific information that is used to uniquely identify the resource and to identify the tenant that owns the resource. In step 703, the distributed computing network determines that it needs to enter a disaster recovery mode, such as when a software, hardware, or network failure occurs.

[0048] In step 704, the resource metadata records are recreated by the resource provider using the tagged information. In step 705, during disaster recovery, the central service uses the subscriptions and specific information to rebuild its subscription and resource metadata information.

[0049] The resource provider can maintain a list of consumer subscriptions and resources created by the consumers. The resource provider can expose the list of consumer subscriptions with access to the resource provider. The resource provider can provide a list of resources owned by each consumer subscription.

[0050] FIG. 8 is a flowchart illustrating an example method for recovery from a failure within a distributed computer network. In step 801, the central service synchronizes with the resource providers to update the central service's resource and subscription information. In step 802, once the central service recovers from the failure, the central service is reconfigured to ensure it has the latest resource provider information. In step 803, all registered resource providers are contacted by the central service to synchronize the tenant subscriptions and resources.

[0051] In step 804, the resource providers expose the resource and subscription information. In step 805, the central service tags the resources have been used for viewing by the service provider or tenant.

[0052] During disaster recovery, the central service can synchronize with other services, such as a billing system that approves subscription creation. Subscriptions not in the billing system can be suspended in the recovered system. When information about a subscription is missing and the information is not possible to recover automatically, the service provider can be contacted or requested to initiate a manual recovery. The manual recovery can require creating a new subscription. The manual recovery can require moving resources from the recovered subscription to the newly recreated subscription at the time of verification of resource ownership.

[0053] An example computer-implemented method for recovery from a disaster within a cloud computing infrastructure includes creating, by a resource provider, resources on a distributed computing network, the resources each having a metadata record; tagging, by the resource provider, the resources with specific information, the specific information being used to uniquely identify the resources and to identify a tenant that owns the resources; determining when the distributed computing network is in a disaster recovery mode; and recreating, by the resource provider, the resource metadata records using the tagged information.

[0054] Additional embodiments of the method further include maintaining, by the resource provider, a list of consumer subscriptions and resources created by a consumer.

[0055] Additional embodiments of the method further include exposing, by the resource provider, a list of consumer subscriptions with access to the resource provider.

[0056] Additional embodiments of the method further include providing, by the resource provider, a list of resources owned by each consumer subscription.

[0057] Additional embodiments of the method further include using, by the central service during disaster recovery, the subscriptions and specific information to rebuild its subscription and resource metadata information.

[0058] Another example computer-implemented method for recovery from a failure within a distributed computer network includes synchronizing a central service with resource providers to update the central service's resource and subscription information; reconfiguring the central service and ensuring it has the latest resource provider information once the central service recovers from the failure; contacting, by the central service, all registered resource providers to synchronize tenant subscriptions and resources; exposing, by the resource provider, resource and subscription information; and tagging, by the central service, recovered resources for review by a service provider or tenant.

[0059] Additional embodiments of the method further include synchronizing the central service with other services.

[0060] In other embodiments of the method, the other services include a billing system that approves subscription creation.

[0061] Additional embodiments of the method further include suspending subscriptions that are not in the billing system.

[0062] In other embodiments of the method, when information about a subscription is missing and the information is not possible to be recovered automatically, a service provider is requested to initiate a manual recovery.

[0063] In other embodiments of the method, the manual recovery requires creating a new subscription.

[0064] In other embodiments of the method, at the time of verification of resource ownership, the manual recovery requires moving resources from the recovered subscription to a newly recreated subscription.

[0065] An example system includes a distributed computing network that executes a central service that manages user subscriptions and resource providers, a plurality of resource providers each configured to create resources as required by user subscriptions, the resources each having a metadata record, the resources being tagged by the resource providers with specific information that uniquely identifies the resource and identifies the user that owns the resource, wherein during recovery from a failure of the distributed computing network, the resources recreate the metadata record using the tagged information.

[0066] In additional embodiments of the system, the central service synchronizes with the resource providers to update resource and subscription information on the central service.

[0067] In additional embodiments of the system, once the central service recovers from a failure, the central service contacts all registered resource providers to synchronize tenant subscriptions and resources.

[0068] In further embodiments of the system, the resource providers expose resource and subscription information.

[0069] In additional embodiments of the system, the central service tags the resources for review by a service provider or tenant upon recovery from a failure.

[0070] In additional embodiments of the system, during recovery from a failure, the central service synchronizes with a billing system that approves subscription creation.

[0071] Additional embodiments of the system include a data store that includes user subscription and resource information.

[0072] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for recovery from a failure of a resource provider within a distributed computer network, the method performed by the resource provider, the method comprising: registering the resource provider with a central service; associating a resource on the distributed computer network with particular information that uniquely identifies the resource and enables the resource provider to determine a resource owner; and in response to the failure of the resource provider: scanning the resource and the particular information associated with the resource; recreating a metadata record in a metadata repository based at least in part on the particular information, wherein the metadata record and the particular information are different; synchronizing the resource with the central service to obtain updated subscription information from the central service; and updating subscription information in the metadata record based at least in part on the updated subscription information obtained from the central service.

2. The method of claim 1, wherein the particular information associated with a particular resource enables a metadata record corresponding to the particular resource to be recreated.

3. The method of claim 1, further comprising: detecting that at least one recreated metadata record violates a business rule; and flagging the at least one recreated metadata record for manual action.

4. The method of claim 1, further comprising exposing to the central service consumer subscriptions having access to the resource provider and one or more resources owned by each consumer subscription.

5. The method of claim 1, further comprising: in response to the failure of the resource provider, restoring the resource provider from a backup.

6. The method of claim 1, further comprising: creating the resource on behalf of a consumer; and maintaining metadata about the resource.

7. A method for recovery from a failure of a central service within a distributed computer network, the method performed by the central service, the method comprising: re-registering a plurality of resource providers, wherein the plurality of resource providers enable consumers to create resources, and wherein the plurality of resource providers have previously registered with the central service prior to the failure of the central service; receiving resource information and subscription information from the plurality of resource providers; during recovery of the central service, recreating metadata records associated with the resources using the resource information and the subscription information received from the plurality of resource providers, wherein the metadata records are different from the resource information and the subscription information; and routing cloud consumer requests through the central service to the plurality of resource providers that have re-registered with the central service; synchronizing the metadata records at the central service with corresponding metadata records maintained by the plurality of resource providers.

8. The method of claim 7, further comprising verifying that the central service is able to communicate with the plurality of resource providers.

9. The method of claim 7, further comprising: in response to the failure of the central service, restoring the central service from a backup.

10. The method of claim 7, further comprising: when information about a subscription is lost and cannot be automatically recovered, requesting a service provider to initiate manual recovery.

9. A method for recovery from a failure of a central service within a distributed computer network, the method performed by the central service, the method comprising: re-registering a plurality of resource providers, wherein the plurality of resource providers enable consumers to create resources, and wherein the plurality of resource providers have previously registered with the central service prior to the failure of the central service; receiving resource information and subscription information from the plurality of resource providers; during recovery of the central service, recreating metadata records associated with the resources using the resource information and the subscription information received from the plurality of resource providers, wherein the metadata records are different from the resource information and the subscription information; and routing cloud consumer requests through the central service to the plurality of resource providers that have re-registered with the central service; synchronizing the metadata records at the central service with corresponding metadata records maintained by the plurality of resource providers.

10. The method of claim 9, further comprising verifying that the central service is able to communicate with the plurality of resource providers. in response to the failure of the central service, restoring the central service from a backup. when information about a subscription is lost and cannot be automatically recovered, requesting a service provider to initiate manual recovery.

11. The method of claim 10, wherein the manual recovery requires creation of new subscriptions.

12. The method of claim 10, wherein the manual recovery requires moving resources from a recovered subscription to a newly recreated subscription upon verifying resource ownership.

13. The method of claim 7, further comprising: synchronizing the central service with a billing system that approves subscription creation; and suspending any newly created subscriptions that are not included in the billing system.

14. The method of claim 7, wherein the distributed computer network enters a disaster recovery mode in response to the failure of the central service.

15. A system for recovering from a failure of a central service within a distributed computer network, comprising: one or more processors; memory in electronic communication with the one or more processors; and instructions stored in the memory, the instructions being executable by the one or more processors to: re-register a plurality of resource providers, wherein the plurality of resource providers enable consumers to create resources, and wherein the plurality of resource providers were previously registered with the central service prior to the failure of the central service; receive resource information and subscription information from a plurality of resource providers; recreate metadata records associated with the resources using the resource information and the subscription information received from the plurality of resource providers during a recovery of the central service, wherein the metadata records are different from the resource information and the subscription information; and route cloud consumer requests through the central service to the plurality of resource providers that have re-registered with the central service; synchronize the metadata records at the central service with corresponding metadata records maintained by the plurality of resource providers.

16. The system of claim 15, further comprising additional instructions executable by the one or more processors to verify that the central service is able to communicate with the plurality of resource providers.

17. The system of claim 15, further comprising additional instructions executable by the one or more processors to recover the central service from a backup in response to the failure of the central service.

18. The system of claim 15, further comprising additional instructions executable by the one or more processors to request a service provider to initiate a manual recovery when information about a subscription is lost and the information cannot be automatically recovered.

Images