Logical deletion of data in a sharded database

Abstract

Methods, systems, and computer program products are provided for a sharded database. The method includes receiving a request to delete a specified document from a primary shard of the sharded database. The method further includes inserting a logical delete document identifying the specified document into the logical delete shard, where the specified document remains in the primary shard. The method further includes receiving a query from a client application, where the specified document satisfies the query. The method further includes preventing the specified document from being returned in response to the query while a logical delete document associated with the specified document remains in the logical delete shard.

Description

[Technical Field]

[0001] The present invention generally relates to methods, systems, and computer program products for sharded databases, and more particularly, to methods, systems, and computer program products for soft deletion of data in a sharded database. [Background technology]

[0002] Modern database systems provide fast information storage, searching, and retrieval capabilities. However, the amount of digital content is growing at an exponential rate, requiring significant storage systems to store and manage the content. Therefore, modern databases often interact with or are part of computer applications that collect, update, analyze, or report on large data sets.

[0003] When these data sets are so large and the demand for accessing them so high that it reaches the performance or storage thresholds of a single server, the data can be distributed across multiple servers to provide additional performance and storage capacity. Each segment in such a distributed database system is known as a "shard." The functions of individual shards can be assigned according to a strategy designed to balance user load and optimize database system performance. By distributing, or "sharding," the database in this way, performance and storage limitations can be overcome. Summary of the Invention

[0004] This exemplary embodiment provides logical deletion of data in a sharded database. One embodiment includes receiving a request to delete a specified document from a primary shard of the sharded database. The embodiment further includes inserting a logical delete document identifying the specified document into the logical delete shard, where the specified document remains in the primary shard. The embodiment further includes receiving a query from a client application, where the specified document satisfies the query. The embodiment further includes preventing the specified document from being returned in response to the query while the logical delete document associated with the specified document remains in the logical delete shard. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of this embodiment. Such embodiments enable logical deletion and restoration of primary data without any write or delete operations on the primary shard and without any changes to the primary shard's index, thus eliminating such time-consuming processes and resulting in significant performance improvements over previous logical delete processes in NoSQL databases.

[0005] In another embodiment, the embodiment further includes receiving a restore request to restore the specified document to the sharded database, and restoring the specified document to the sharded database in response to the restore request, where restoring includes removing the logically deleted document from the logically deleted shard. Thus, this embodiment enables restoration of logically deleted data in response to the restore request, thereby providing expected functionality and avoiding costly data loss for end users accustomed to having data undo or recovery capabilities.

[0006] In another embodiment, the embodiment further includes detecting when the time since receiving the request to delete the specified document reaches a defined retention period and performing a hard deletion process to purge the specified document from the sharded database. Thus, the embodiment allows a period of time to restore the data, after which such data is permanently removed to prevent unnecessary data from consuming database resources.

[0007] One embodiment includes a computer-usable program product that includes a computer-readable storage medium and program instructions stored on the storage medium.

[0008] One embodiment includes a computer system including a processor, a computer-readable memory, a computer-readable storage medium, and program instructions stored on the storage medium for execution by the processor via the memory.

[0009] The appended claims set forth the novel features which are believed characteristic of the invention. However, the invention itself and its preferred mode of use, together with its objects and advantages, will best be understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram of a network of data processing systems in which illustrative embodiments may be implemented; [Figure 2] 1 is a block diagram of a data processing system in which illustrative embodiments may be implemented; [Figure 3] FIG. 1 is a block diagram of an exemplary shard system in accordance with an exemplary embodiment. [Figure 4] FIG. 1 is a block diagram of a sharded database in accordance with an example embodiment; [Figure 5] FIG. 1 is a block diagram of a sharded database in accordance with an example embodiment; [Figure 6] FIG. 1 is a block diagram of an exemplary sharded database in accordance with an exemplary embodiment. [Figure 7] FIG. 1 is a block diagram of an exemplary sharded database in accordance with an exemplary embodiment. [Figure 8] FIG. 1 is a block diagram of an exemplary sharded database in accordance with an exemplary embodiment. [Figure 9] FIG. 1 is a block diagram of an exemplary sharded database in accordance with an exemplary embodiment. [Figure 10] FIG. 1 is a block diagram of a timeline progression of an SDS index building process according to an example embodiment. [Figure 11] 1 is a flowchart of an exemplary process for logical deletion of data in a sharded database, in accordance with an exemplary embodiment. [Figure 12A] FIG. 1 illustrates a first portion of a flowchart of an exemplary process for logical deletion of data in a sharded database, in accordance with an exemplary embodiment. [Figure 12B] FIG. 10 illustrates a second portion of a flowchart of an exemplary process for logical deletion of data in a sharded database, in accordance with an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Modern databases include cross-platform, document-oriented NoSQL (Not only Structured Query Language) databases. Such databases eschew traditional table-based relational database structures in favor of sharding. When a rapidly growing database runs on a single server, the database will eventually reach the limits of the computing resources the server can provide. These limits include capacity limits for storing data and processing limits for handling queries and other database commands.

[0012] Sharding involves dividing data into two or more subsets of data stored on separate servers. Herein, we refer to such subsets of data as "primary shards" or "primary data shards." This distributed architecture can support the deployment and high-throughput operation of very large data sets. Thus, sharding helps achieve a scalable setup for storing large amounts of data across many primary shards, each on its own server.

[0013] A sharded database may also include a data replication scheme in which the database generates a set of copies of each shard, each holding the same data. At any given time, only one shard in the replication set acts as the primary shard, and all other replication shards are secondary shards. All write and read operations are directed to the primary shard and then (as needed) evenly distributed to other secondary shards in the set.

[0014] Despite these advantages that sharding offers in terms of scalability, the problem remains that the process of adding data to or removing data from a primary shard is a relatively time-consuming process that negatively impacts database performance. When adding or removing data from a primary shard, the primary index must be rebuilt to update the index records to reflect the change. Indexes are used to access the database, and large databases can require several large indexes that must be maintained to efficiently access the database. Indexes typically need to be maintained or updated when the database is changed. Rebuilding such indexes can take a long time and causes the index to become unavailable for queries until the index update is complete.

[0015] This problem is exacerbated by past attempts to implement a logical delete process within a sharded database. When data is logically deleted, it cannot be selected or used, but it can be restored using the normal available functionality of the database or server. In contrast, without logical delete, data is always hard deleted, which means the data is permanently lost and cannot be restored without extraordinary effort that may or may not be effective.

[0016] Data loss can be very costly and frustrating for end users who are accustomed to having the ability to undo or resurrect data. Therefore, implementing logical delete functionality is important to prevent data loss and provide users with the functionality they expect. However, implementing logical delete functionality is a system-specific task that presents unique challenges for some types of systems that are not an issue in other types of systems. For example, past efforts to implement logical delete functionality in document-oriented NoSQL databases have negatively impacted database performance. This is because the previous logical delete process involved writing a flag or other such data to the primary shard to mark the "logically deleted" data as unavailable; when the "logically deleted" data is purged, both the "logically deleted" data and the flag are deleted; and each of these write and delete operations requires an index update operation on the shard index. However, write, delete, and index update operations are costly for the database to perform in terms of time and processing. Therefore, these past types of logical deletion techniques present a technical problem in that they require several additional operations that are costly and require an undesirable trade-off of reduced performance in exchange for the benefit of restorable data.

[0017] To solve these technical problems, the disclosed embodiments introduce an additional shard called a logical delete shard (SDS) that stores logically deleted documents (SDDs) that reference logically deleted primary data. Unlike secondary shards, which maintain replicas of data from a primary shard regardless of the status of the primary shard data, an SDS stores only documents related to logically deleted primary shard documents that have not yet been purged. During this period, the referenced primary documents remain in the primary shard unless and until they are purged (i.e., physically deleted). If logically deleted data is restored instead of purged, this restoration involves removing the associated SDD from the SDS. This enables the logical deletion and restoration of primary data without any write or delete operations on the primary shard and without any changes to the primary shard's index. Eliminating these time-consuming processes results in significant performance improvements over previous logical deletion processes in NoSQL databases.

[0018] An example embodiment of a sharded database application including logical delete receives a request to delete a specified document from a primary shard of a sharded database, and in response inserts an SDD identifying the specified document into the SDS. The specified document remains in the primary shard. If the application subsequently receives a query from a client application such that the specified document satisfies the query, the application prevents the specified document from being returned with the query results as long as the SDD associated with the specified document remains in the SDS.

[0019] In an exemplary embodiment, a database application performs operations on a NoSQL database. Embodiments include any of a wide range of technologies and architectures, including NoSQL. For example, in some embodiments, the NoSQL database is a document-oriented database or document store that stores data in the form of documents, each document having a unique identifier (UID), which is some kind of metadata that provides a degree of structure to the data. The UIDs can be formatted according to any of a variety of known data formats, and the document data is stored in the desired format. In some embodiments, a number of database servers collectively provide the services of the NoSQL database. Thus, in some embodiments, the NoSQL database comprises a widely distributed, non-relational database system that enables rapid, ad-hoc organization and analysis of extremely large volumes of heterogeneous data types. In some embodiments, the NoSQL database comprises a database referred to as a cloud database, a non-relational database, a big data database, or any of a myriad of other terms for NoSQL databases, or any combination thereof.

[0020] Embodiments of the SDD may reference the specified document in any of a variety of different ways. For example, in some embodiments, the SDD identifies a particular document by including a unique identifier (UID) for the specified document. Alternatively or additionally, in some embodiments, the SDD includes a pointer to the specified document.

[0021] In some embodiments, when an application receives a query from a client application, the database application executes the query against the primary data shard, which then executes the query against the SDS. The application identifies that the query results from the primary shard include logically deleted documents by detecting that an SDD was returned from the query against the SDS. The application can identify the logically deleted documents by evaluating the SDD. Each SDD identifies a unique logically deleted document from the primary shard, so the application uses this information to locate the logically deleted documents in the query results returned from the primary shard. For example, in some embodiments, the SDD includes the UID of the logically deleted document in the primary shard, and the application locates the logically deleted document in the query results by locating the document in the query results that has the UID of the SDD returned from the query against the SDS. After the application locates the logically deleted document in the query results, the application removes the logically deleted document from the query results. By excluding the soft-deleted documents, the query results appear as expected, since the query results do not include soft-deleted documents from the primary shard.

[0022] In an exemplary embodiment, an application builds an index or multiple SDS indexes to facilitate queries against the SDS. In some embodiments, this index building includes associating keys with corresponding SDD locations. In some embodiments, a database application builds one or more indexes for a primary shard.

[0023] There are many different types of indexes that may be used by an application depending on implementation-specific considerations. For example, for unstructured or human language data, a full-text index with a language analyzer to convert text blobs into index entries may be used. For geospatial or geotemporal data, points, polygons, and other shapes in multidimensional space may be indexed. In some embodiments, an application builds one or more SDS indexes to match the number and type of indexes that exist for the primary shard. This allows queries against the SDS to be consistent with queries against the primary shard.

[0024] In an exemplary embodiment, when an application queries the SDS, the application first checks the status of the SDS index to determine whether the SDS index build is complete or incomplete. If the application determines that the SDS index build is complete, the application queries the SDS using the SDS index. Otherwise, if the application determines that the SDS index build is not complete, the application queries the SDS using a full table scan.

[0025] In an exemplary embodiment, after a document in a primary shard is soft-deleted, the document remains soft-deleted and available for restore for a specified retention period. In some embodiments, the specified retention period is a period set by a user. The application periodically checks for soft-deleted documents that remain soft-deleted for a time equal to or longer than the specified retention period. If a soft-deleted document remains soft-deleted for the specified retention period, the application purges the soft-deleted document from the primary shard. In some embodiments, the application purges soft-deleted documents by performing a physical delete process on the specified document from the sharded database. In some such embodiments, the physical delete process includes deleting the specified document from the primary shard of the sharded database, followed by updating the index of the primary shard, followed by updating the soft-delete index in the SDS, and then deleting the SDS identifying the specified document.

[0026] For clarity of explanation, and without implying limitation to the description, the exemplary embodiments are described using several exemplary configurations. From this disclosure, one skilled in the art may devise numerous modifications, adaptations, and variations of the described configurations to achieve the described objectives, which are contemplated within the scope of the exemplary embodiments.

[0027] Furthermore, the figures and exemplary embodiments use simplified diagrams of data processing environments. An actual computing environment may contain additional structures or components not shown or described herein, or structures or components that differ from the structures or components shown but have functionality similar to that described herein, without departing from the scope of the exemplary embodiments.

[0028] Furthermore, with respect to specific components, actual or hypothetical, the exemplary embodiments are described by way of example only, and the steps described by various exemplary embodiments may be adapted to provide explanations for decisions made by, for example, a machine learning classifier model.

[0029] The particular manifestations of these and other similar artifacts are not intended to limit the present invention, and any suitable manifestations of these and other similar artifacts may be selected within the scope of the exemplary embodiments.

[0030] The examples in this disclosure are used for clarity of explanation only and are not intended to limit the exemplary embodiments. The advantages listed herein are merely examples, and are not intended to limit the exemplary embodiments. Additional or different advantages may be realized depending on the particular exemplary embodiment. Furthermore, a particular exemplary embodiment may have some, all, or none of the advantages listed above.

[0031] Additionally, exemplary embodiments may be implemented with any type of data, data source, or access to a data source across a data network. Any type of data storage device may provide data to embodiments of the present invention within the scope of the present invention, whether locally at a data processing system or across a data network. Where embodiments are described using a mobile device, any type of data storage device suitable for use with the mobile device may provide data to such embodiments within the scope of exemplary embodiments, whether locally at the mobile device or across a data network.

[0032] The exemplary embodiments are described using specific code, contrasting descriptions, computer-readable storage media, high-level features, historical data, designs, architectures, protocols, layouts, schematics, and tools as examples only, and are not limited to the exemplary embodiments. Furthermore, in some instances, for clarity of explanation, the exemplary embodiments are described using specific software, tools, and data processing environments as examples only. The exemplary embodiments may be used with other comparable or similar purpose structures, systems, applications, or architectures. For example, other comparable mobile devices, structures, systems, applications, or architectures thereof may be used with such embodiments of the present invention within the scope of the present invention. An exemplary embodiment may be implemented in hardware, software, or a combination thereof.

[0033] The examples in this disclosure are used for clarity of explanation only and do not limit the exemplary embodiments. Additional data, operations, acts, tasks, activities, and processes may be envisioned from this disclosure and are contemplated within the scope of the exemplary embodiments.

[0034] The advantages described herein are examples only and are not intended to limit the exemplary embodiments. Additional or different advantages may be realized depending on the particular exemplary embodiment. Furthermore, a particular exemplary embodiment may have some, all, or none of the advantages listed above.

[0035] Referring now to the figures, and more particularly to Figures 1 and 2, these figures are exemplary illustrations of data processing environments in which illustrative embodiments may be implemented. Figures 1 and 2 are merely examples, and these figures are not intended to assert or imply any limitation with regard to the environments in which different embodiments may be implemented. Particular implementations may make many modifications to the depicted environments based on the following description.

[0036] 1 illustrates a block diagram of a network of data processing systems in which exemplary embodiments may be implemented. Data processing environment 100 is a network of computers in which exemplary embodiments may be implemented. Data processing environment 100 includes network 102. Network 102 is the medium used to provide communications links between various devices and computers connected together within data processing environment 100. Network 102 may include connections such as wired communications links, wireless communications links, or fiber optic cables.

[0037] Client or server are merely examples of roles for certain data processing systems connected to network 102, and client or server is not intended to exclude other configurations or roles for these data processing systems. Coupled to network 102 is data processing system 104. Software applications may be executed on data processing systems within data processing environment 100. Any software applications described as executing within processing system 104 of FIG. 1 may be similarly configured to execute within another data processing system. Any data or information stored within or generated within data processing system 104 of FIG. 1 may be similarly configured to be stored within or generated within another data processing system. A data processing system, such as data processing system 104, may contain data and may have software applications or software tools that execute computing processes thereon. In one embodiment, data processing system 104 includes memory 124, which includes application 105A, which may be configured to perform one or more of the data processor functions described herein according to one or more embodiments.

[0038] Coupled to network 102 are server 106 and storage unit 108. Storage unit 108 includes database 109 configured to store data described herein with respect to various embodiments, such as image data and attribute data. Server 106 is a conventional data processing system. In one embodiment, server 106 includes a processing element of stream processing application 105B, which may be configured to perform one or more of the processor functions described herein according to one or more embodiments.

[0039] Further coupled to network 102 are clients 110, 112, and 114. A conventional data processing system, such as server 106 or client 110, 112, or 114, can contain data and have software applications or tools that execute conventional computing processes on the system.

[0040] Merely by way of example, and without implying limitation to such architecture, Figure 1 illustrates certain components that may be used in an exemplary implementation of an embodiment. For example, server 106 and clients 110, 112, and 114 are illustrated as servers and clients by way of example only, and this does not imply limitation to a client-server architecture. As another example, embodiments may be distributed across several data processing systems and across a data network as shown, while other embodiments may be implemented on a single data processing system within the scope of the exemplary embodiment. Conventional data processing systems 106, 110, 112, and 114 further represent exemplary nodes of clusters, partitions, and other configurations suitable for implementing embodiments.

[0041] Device 132 is an example of a conventional computing device as described herein. For example, device 132 may take the form of a smartphone, a tablet computer, a laptop computer, a fixed or portable client 110, a wearable computing device, or other suitable device. In one embodiment, device 132 sends a request to server 106 to perform one or more data processing tasks, such as tasks that initiate processes as described herein, via stream processing application 105B. Any software application described as executing within another conventional data processing system of FIG. 1 may similarly be configured to execute within device 132. Any data or information stored in or generated within another conventional data processing system of FIG. 1 may similarly be configured to be stored within or generated within device 132.

[0042] Server 106, storage unit 108, data processing system 104, clients 110, 112, and 114, and device 132 may be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connectivity. Clients 110, 112, and 114 may be, for example, personal computers or network computers.

[0043] In the depicted example, server 106 may provide data such as boot files, operating system images, and applications to clients 110, 112, and 114. In this example, clients 110, 112, and 114 may be clients of server 106. Clients 110, 112, 114, or some combination of these clients, may include their own data, boot files, operating system images, and applications. Data processing environment 100 may include additional servers, clients, and other devices not shown.

[0044] In the depicted example, memory 124 may provide data such as boot files, operating system images, and applications to processor 122. Processor 122 may include its own data, boot files, operating system images, and applications. Data processing environment 100 may include additional memory, processors, and other devices not shown.

[0045] In the depicted example, data processing environment 100 may be the Internet. Network 102 may represent a collection of networks and gateways that communicate with each other using Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols. At the heart of the Internet is a backbone of data communication links between major nodes or host computers, including thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, data processing environment 100 may also be implemented as a number of different types of networks, such as, for example, an intranet, a local area network (LAN), or a wide area network (WAN). Figure 1 is intended as an example and is not intended to limit the architecture of different illustrative embodiments.

[0046] Among other uses, the data processing environment 100 can be used to implement a client-server environment in which exemplary embodiments can be implemented. A client-server environment allows software applications and data to be distributed across a network, such that applications function using interactivity between traditional client and server data processing systems. The data processing environment 100 can also use a service-oriented architecture, in which interoperable software components distributed across a network can be packaged together as coherent business applications. The data processing environment 100 can also take the form of a cloud, using a cloud computing model of service delivery to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal administrative effort or interaction with the service provider.

[0047] Referring to Figure 2, this figure illustrates a block diagram of a data processing system in which exemplary embodiments may be implemented. Data processing system 200 is an example of a conventional computer, such as data processing system 104, server 106, or clients 110, 112, and 114 of Figure 1, or another type of device in which computer-usable program code or instructions implementing processes for exemplary embodiments may be located.

[0048] Data processing system 200 further represents a conventional data processing system or configuration thereof, such as conventional data processing system 132 of FIG. 1, in which computer-usable program code or instructions for implementing the processes of the illustrative embodiments may be located. Data processing system 200 is described as a computer by way of example only, but is not limited to a computer. Implementations in the form of other devices, such as device 132 of FIG. 1, may modify data processing system 200, such as by adding a touch interface, and data processing system 200 may also exclude certain illustrated components without departing from the general description of the operation and functionality of data processing system 200 described herein.

[0049] In the depicted example, data processing system 200 uses a hub architecture including a north bridge and memory controller hub (NB / MCH) 202 and a south bridge and input / output (I / O) controller hub (SB / ICH) 204. Coupled to north bridge and memory controller hub (NB / MCH) 202 are processing unit 206, main memory 208, and graphics processor 210. Processing unit 206 may include one or more processors and may be implemented using one or more heterogeneous processor systems. Processing unit 206 may be a multi-core processor. In certain implementations, a graphics processor 210 may be coupled to NB / MCH 202 through an accelerated graphics port (AGP).

[0050] In the illustrated example, a local area network (LAN) adapter 212 is coupled to a south bridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a universal serial bus (USB) and other ports 232, and PCI / PCIe devices 234 are coupled to the south bridge and I / O controller hub 204 via a bus 238. A hard disk drive (HDD) or solid-state drive (SSD) 226 and a CD-ROM 230 are coupled to the south bridge and I / O controller hub 204 via a bus 240. PCI / PCIe devices 234 may include, for example, an Ethernet adapter, an add-in card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. The ROM 224 may be, for example, a flash binary input / output system (BIOS). The hard disk drive 226 and CD-ROM 230 may use, for example, an Integrated Drive Electronics (IDE), a Serial Advanced Technology Attachment (SATA) interface, or variations thereof, such as external SATA (eSATA) and micro SATA (mSATA). A super I / O (SIO) device 236 may be coupled to the south bridge and I / O controller hub (SB / ICH) 204 via a bus 238.

[0051] Memory such as main memory 208, ROM 224, or flash memory (not shown) are some examples of computer-usable storage devices. Hard disk drives or solid-state drives 226, CD-ROMs 230, and other similarly usable devices are some examples of computer-usable storage devices that include computer-usable storage media.

[0052] An operating system runs on processing unit 206. The operating system coordinates and provides control of various components within data processing system 200 of FIG. 2. The operating system may be a commercially available operating system for any type of computing platform, including, but not limited to, server systems, personal computers, and mobile devices. An object-oriented programming system or other type of programming system may run in conjunction with the operating system, and these programming systems may provide calls to the operating system from programs or applications running on data processing system 200.

[0053] Instructions for an operating system, an object-oriented programming system, and an application or program, such as application 105 of Figure 1, may be located on a storage device, for example in the form of code 226A on hard disk drive 226, and loaded into at least one of one or more memories, such as main memory 208, for execution by processing unit 206. The processes of the exemplary embodiments may be performed by processing unit 206 using computer-implemented instructions, which may be located in a memory, such as main memory 208, read-only memory 224, or one or more peripheral devices.

[0054] Additionally, in one case, code 226A can be downloaded across network 201A from remote system 201B, where similar code 201C is stored on storage device 201D of remote system 201B. In another case, code 226A can be downloaded across network 201A to remote system 201B, where downloaded code 201C is stored on storage device 201D of remote system 201B.

[0055] The hardware in Figures 1-2 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, may be used in addition to or in place of the hardware depicted in Figures 1-2. Furthermore, the processes of the illustrative embodiments may be applied to multiprocessor data processing systems.

[0056] In some illustrative examples, data processing system 200 may be a personal digital assistant (PDA), which is typically configured with flash memory to provide non-volatile memory for storing operating system files and / or user-generated data. The bus system may include one or more buses, such as a system bus, an I / O bus, and a PCI bus. Of course, the bus system may be implemented using any type of communications fabric or architecture that provides for data transfer between different components or devices connected to the fabric or architecture.

[0057] The communications unit may include one or more devices, such as a modem or network adapter, used to transmit and receive data. The memory may be, for example, main memory 208 or a cache, such as that found in north bridge and memory controller hub 202. The processing unit may include one or more processors or CPUs.

[0058] 1-2 and described above are not intended to imply architectural limitations. For example, in addition to taking the form of a mobile or wearable device, data processing system 200 may also be a tablet computer, a laptop computer, or a telephone.

[0059] When a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates in the manner of data processing system 200 using virtualized representations of some or all of the components depicted in data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 is represented as a virtualized instance of all or a portion of hardware processing unit 206 available in the host data processing system, main memory 208 is represented as a virtualized instance of all or a portion of main memory 208 that may be available in the host data processing system, and disk 226 is represented as a virtualized instance of all or a portion of disk 226 that may be available in the host data processing system. The host data processing system in such cases is represented by data processing system 200.

[0060] Referring to Figure 3, this figure shows a block diagram of an exemplary sharded system 300 according to one example embodiment. In the illustrated embodiment, sharded system 300 includes a sharded database 302 that allows one or more client applications 322 to access data items distributed across multiple primary database shards 312, 314, 316, and 318. In one embodiment, sharded database 302 is an example of database 109 in Figure 1, and client application 322 is an example of application 105A / 105B in Figure 1.

[0061] Sharded system 300 includes client applications 322 and primary database shards 312, 314, 316, and 318. The amount of client applications 322 and primary database shards 312, 314, 316, and 318 in system 300 can vary. In some embodiments, each of primary database shards 312, 314, 316, and 318 is a separate, independent database that does not need to be aware of other shards in system 300. In some such embodiments, each of primary database shards 312, 314, 316, and 318 comprises, for example, a separate database server and relational database. In some embodiments, there are two or more client applications 322 that are on separate computing systems and operate independently of each other among the clients 322. In some such embodiments, each of the clients 322 runs a separate instance of a software program that utilizes a hash function to calculate the identity of a particular shard among the primary database shards 312, 314, 316, and 318 where or which stores a particular data item based on the primary key of that particular data item. In some embodiments, such data items are separate records having different values ​​for a similar set of attributes.

[0062] Generally, to perform an operation on a data item already stored in a particular one of primary database shards 312, 314, 316, and 318, client application 322 executes one or more instructions for that data item by issuing a database command to database 302. For example, such instructions may include query commands processed by one or more primary query engines 308 or delete commands processed by deletion manager 304. In some embodiments, database 302 includes a primary query engine 308 for each primary database shard 312-318. Example embodiments further provide a restore command processed by restore manager 324 that enables deleted data to be restored within a specified period of time after the data was deleted.

[0063] According to one embodiment of the present invention, database 302 is able to restore deleted data because system 300 includes logically deleted shard (SDS) 320, which stores logically deleted documents containing the logical delete status of documents deleted from primary database shards 312, 314, 316, and 318. After a designated period of time, the logically deleted documents are removed from SDS 320 by purge manager 306 as part of a physical delete process that purges the deleted data. During the designated period of time, SDS query engine 310 executes queries of SDS 320 corresponding to the queries executed by primary query engine 308 for each of primary database shards 312, 314, 316, and 318. Because the same query is performed against both the collection of primary database shards 312, 314, 316, and 318 and SDS 320, and because documents in SDS 320 are essentially copies of or reference documents from the logically deleted primary database shards 312, 314, 316, and 318, when SDS query engine 310 finds a document in SDS 320, it means that if a query matches a document from SDS 320, that same query will also match documents from the logically deleted primary database shards 312, 314, 316, and 318. Therefore, database 302 removes from the primary shard query results any query results identified by SDS 320 documents returned as query results. This action removes the logically deleted documents from the search results, presumably because deleted documents are not expected to be included in the query results.

[0064] Referring to Figure 4, this figure shows a block diagram of a sharded database 400 according to an example embodiment. In one embodiment, sharded database 400 is an example of database 302 of Figure 3 or database 109 of Figure 1.

[0065] In the illustrated embodiment, database 400 includes a deletion manager 402, a purge manager 404, a query subtractor 406, a query manager 408, primary database shards 410, 412, and 414, an SDS 416, an index manager 418, index engines 420, 422, and 424, an index synchronizer 426, SDS indexes 428, 430, and 432, a query aggregator 434, primary query engines 436, 438, and 440, an SDS query engine 442, and a restore manager 444. In some embodiments, the functionality described herein is distributed across multiple systems, which may include a combination of software and / or hardware-based systems, such as application-specific integrated circuits (ASICs), computer programs, or smart phone applications. In one embodiment, deletion manager 402 is an example of deletion manager 304, purge manager 404 is an example of purge manager 306, primary database shards 410, 412, and 414 are examples of primary database shards 312, 314, 316, and 318, SDS 416 is an example of SDS 320, primary query engines 436, 438, and 440 are examples of primary query engine 308, SDS query engine 442 is an example of SDS query engine 310, and restore manager 444 is an example of restore manager 324.

[0066] In the illustrated embodiment, index manager 418 dispatches index requests to different primary shards 410, 412, and 414 by invoking corresponding respective index engines 420, 422, and 424. Index engines 420, 422, and 424 execute the index requests. While one index engine 420, 422, and 424 is shown for each primary shard 410, 412, and 414, alternative embodiments include multiple index engines for each primary shard. In some such embodiments, database 400 builds one or more indexes for primary shards 410, 412, and 414. There are many different types of indexes that may be used by an application depending on implementation-specific considerations. For example, for unstructured data or human language data, a full-text index with a language analyzer to convert text blobs into index entries may be used. For geospatial or geotemporal data, points, polygons, and other shapes in multidimensional space may be indexed. In some such embodiments, database 400 includes a different type of index engine for each index type.

[0067] In some embodiments, an index synchronizer 426 asynchronously builds and updates one or more SDS indexes 428, 430, and 432 for the SDS 416 to match the number and type of indexes that exist for each of the primary shards 410, 412, and 414. This allows queries to the SDS 416 to be consistent with queries to the primary shards 410, 412, and 414.

[0068] In some embodiments, when query manager 408 receives a query from a client application, query manager 408 instructs primary query engines 436, 438, and 440 to execute the query against primary data shards 410, 412, and 414, and instructs SDS query engine 442 to execute the query against SDS 416. Query aggregator 434 receives the query results from each of primary query engines 436, 438, and 440 and performs result aggregation to combine the results into a single aggregated result set.

[0069] The query subtractor 406 receives the query results found by the SDS query engine 442. The query subtractor 406 recognizes that the aggregated result set from the primary shards 410, 412, and 414 includes logically deleted documents by detecting that an SDD was returned from a query to the SDS 416. The query subtractor 406 can identify logically deleted documents among the documents in the aggregated result set by evaluating the SDD. Each SDD identifies a unique logically deleted document from the primary shards 410, 412, and 414, and the query subtractor 406 uses this information to locate the logically deleted documents in the aggregated result set.

[0070] For example, in some embodiments, the SDD includes the UIDs of the logically deleted documents in the primary shards 410, 412, and 414, and the query subtractor 406 locates the logically deleted documents in the aggregated result set by locating documents in the query results that have the UIDs of the SDDs returned from a query on the SDS 416. After the query subtractor 406 locates the logically deleted documents in the query results, the query subtractor 406 removes the logically deleted documents from the query result set. Because this query result set does not include the logically deleted documents from the primary shards 410, 412, and 414, excluding the logically deleted documents ensures that the query result set appears as expected. The purge manager performs physical deletes on the primary data and indexes in the primary and logically deleted shards.

[0071] In an exemplary embodiment, after a document in primary shards 410, 412, and 414 is soft-deleted, the document remains soft-deleted and is available for restoration by restore manager 444 for a specified retention period. In some embodiments, the specified retention period is a period set by a user. The application periodically checks for soft-deleted documents that have remained soft-deleted for a time equal to or greater than the specified retention period. If a soft-deleted document remains soft-deleted for the specified retention period, purge manager 404 purges the soft-deleted document from primary shards 410, 412, and 414. In some embodiments, purge manager 404 purges soft-deleted documents by performing a physical delete process on the specified document from sharded database 400. In some such embodiments, the physical delete process involves deleting the specified document from a primary shard 410, 412, or 414 of the sharded database 400, followed by updating the indexes of the primary shards 410, 412, and 414 by the index engines 420, 422, and 424, followed by updating the logical delete indexes 428, 430, and 432 of the SDS 416, followed by deleting the SDS that identifies the specified document.

[0072] Referring to Figure 5, this figure shows a block diagram of a sharded database 500 according to an example embodiment. In one embodiment, sharded database 500 is an example of sharded database 400 of Figure 4, database 302 of Figure 3, or database 109 of Figure 1.

[0073] In the illustrated embodiment, the database 500 includes primary shards 502-516 and an SDS 518. The primary shards 502-516 each include an equal number of addressable data stores. In the illustrated embodiment, the primary shards 502-516 each include 0x1F addresses, although this amount may vary. It is desirable to maintain a balance in the distribution of data across the primary data shards 502-516. One reason this is desirable is because if the shards are unbalanced (e.g., primary shard 502 near capacity and primary shard 504 near empty), RAM and disk space may be underutilized for one primary shard and overutilized for another primary shard. Some embodiments attempt to keep primary indexes in RAM with recently used data to improve execution speed. In an unbalanced situation, the RAM of an overloaded shard may begin to evict data set items or indexes from RAM. Therefore, it is desirable to maintain a balanced data level across the primary shards 502-516.

[0074] On the other hand, the SDS 518 stores data in a unique environment (it stores only logically deleted data). Thus, in some embodiments, the database 500 includes shard balancing to distribute data evenly across the primary shards 502-516, but excludes the SDS 518 from this balancing. Similarly, in some embodiments, the shards 502-516 include both primary and secondary data shards (e.g., when secondary shards are used for data replication). In such embodiments, shard balancing is performed to distribute data evenly across the primary and secondary shards 502-516, but excludes the SDS 518 from this shard balancing.

[0075] Referring to Figure 6, this figure shows a block diagram of an exemplary sharded database 600 in accordance with one exemplary embodiment. More particularly, Figure 6 illustrates the effect of a delete command on a document (DOCB) in sharded database 600. In particular embodiments, sharded database 600 is an example of sharded database 500 of Figure 5, sharded database 400 of Figure 4, database 302 of Figure 3, or database 109 of Figure 1.

[0076] In the illustrated embodiment, sharded database 600 includes three primary shards: Database Shard A (DBSA) 602, Database Shard B (DBSB) 604, and Database Shard C (DBSC) 606. The sharded database 600 receives a delete command to delete a DOCB in DBSB 604. This is illustrated in FIG. 6 by showing the document DOCB with a strikethrough, indicating that the DOCB has been logically deleted as a result of the delete command. However, the strikethrough of the DOCB in DBSB 604 is symbolic and for illustrative purposes, as the delete or restore command does not change the actual document DOCB in DBSB 604. Additionally, as a result of this delete command, an SDD for the DOCB is added to SDS 608. The SDS index is then updated to reflect that the SDD for the DOCB has been added to SDS 608. If the query manager 610 receives a query that is satisfied by the DOCB while the DOCB is in a logically deleted state, the DOCB is returned as a query result along with the other query results. However, because the SDD for the DOCB is in the SDS 608, the DOCB is removed while still in the pre-result. Thus, the final result 612 excludes the logically deleted document DOCB.

[0077] Referring to Figure 7, this figure shows a block diagram of an exemplary sharded database 700 according to one example embodiment. More specifically, Figure 7 shows the effect of a restore command on a previously logically deleted document (DOCB). In certain embodiments, sharded database 700 is an example of sharded database 500 of Figure 5, sharded database 400 of Figure 4, database 302 of Figure 3, or database 109 of Figure 1.

[0078] In the illustrated embodiment, sharded database 700 includes three primary shards: Database Shard A (DBSA) 702, Database Shard B (DBSB) 704, and Database Shard C (DBSC) 706. DBSB 704 shows document DOCB with a dashed strikethrough, indicating that the DOCB has been restored after being previously logically deleted. However, the strikethrough of the DOCB in DBSB 704 is merely symbolic for illustrative purposes, as the delete or restore commands do not change the actual document DOCB in DBSB 704. Instead, the actual changes caused by the restore command are to the SDS 708. As a result of the restore command, the DOCB SDD and the SDS index for the DOCB SDD are removed. Therefore, future queries to query manager 710 that return the DOCB will include the DOCB in their previous and final results.

[0079] Referring to Figure 8, this figure shows a block diagram of an exemplary sharded database 800 according to one example embodiment. More particularly, Figure 8 shows the effect of a query command on logically deleted documents (DOCBs) in sharded database 800. In particular embodiments, sharded database 800 is an example of sharded database 500 of Figure 5, sharded database 400 of Figure 4, database 302 of Figure 3, or database 109 of Figure 1.

[0080] In the illustrated embodiment, sharded database 800 includes three primary shards: database shard A (DBSA) 802, database shard B (DBSB) 804, and database shard C (DBSC) 806. DBSB 804 shows document DOCB with a strikethrough, indicating that the DOCB has been logically deleted. However, because a delete or restore command does not change the actual document DOCB in DBSB 804, the strikethrough of the DOCB in DBSB 804 is merely symbolic for illustrative purposes. Thus, while the DOCB is in a logically deleted state, SDS 808 includes an SDD for the DOCB. When query manager 810 receives a query that is satisfied by the DOCB, the DOCB is returned as a query result along with other query results. However, because the SDD for the DOCB is in SDS 808, the DOCB is removed while still in the previous results. Thus, final result 812 excludes the logically deleted document DOCB.

[0081] Referring to Figure 9, this figure shows a block diagram of an exemplary sharded database 900 in accordance with one example embodiment. More particularly, Figure 9 illustrates the effect of a physical delete or purging operation on previously logically deleted documents (DOCBs). In particular embodiments, sharded database 900 is an example of sharded database 500 of Figure 5, sharded database 400 of Figure 4, database 302 of Figure 3, or database 109 of Figure 1.

[0082] In the illustrated embodiment, sharded database 900 includes three primary shards: Database Shard A (DBSA) 902, Database Shard B (DBSB) 904, and Database Shard C (DBSC) 906. DBSB 904 shows document DOCB with a strikethrough, indicating that purge manager 910 permanently deleted the DOCB after a defined retention period has elapsed since the DOCB was logically deleted. Next, purge manager 910 updates the index of the primary shard to remove the reference to DOCB.Field5. Next, purge manager 910 updates the logical delete index of SDS 908. Finally, purge manager 910 removes SDD from SDS 908, which identifies the purged document DOCB.

[0083] Referring to Figure 10, this figure illustrates a block diagram of a timeline progression of the SDS index construction process according to one example embodiment. In a particular embodiment, the process illustrated in Figure 10 is performed by index synchronizer 1010, which is an example of index synchronizer 426 of Figure 4.

[0084] In an exemplary embodiment, to support efficient execution of SDS queries, the index synchronizer 1010 builds multiple SDS indexes 1012, 1014, and 1016 over the time period T1 through T4. If the SDS indexes 1012, 1014, and 1016 are not complete (i.e., before T4), the database must perform a full table scan (i.e., scan all documents in the SDS) to select an SDD that matches the query statement. If the SDS indexes 1012, 1014, and 1016 are complete (i.e., after T4), the database uses the indexes 1012, 1014, and 1016 to limit the number of SDDs examined to select an SDD that matches the query statement.

[0085] In some embodiments, the index synchronizer 1010 builds multiple indexes for the SDS, each of which is a different index type. In the illustrated embodiment, the index synchronizer 1010 asynchronously builds indexes 1012, 1014, and 1016. Starting at time T1, from index state 1002, the index synchronizer 1010 builds the SDS index 1012, which is completed at time T2, achieving index state 1004. At time T2, from index state 1004, the index synchronizer 1010 builds the SDS index 1014, which is completed at time T3, achieving index state 1006. At time T3, from index state 1006, index synchronizer 1010 builds SDS index 1016, which is completed at time T4, completing index state 1008. In some embodiments, index synchronizer 1010 builds SDS indexes 1012, 1014, and 1016 to match the number and types of indexes that exist for the primary shard, allowing queries against the SDS indexes to be consistent with queries against the primary shard.

[0086] 11 , which shows a flowchart of an example process 1100 for logical deletion of data in a sharded database, according to an example embodiment. In some embodiments, database 302, sharded database 400, sharded database 500, sharded database 600, sharded database 700, sharded database 800, or sharded database 900 performs process 1100.

[0087] In one embodiment, at block 1102, the process receives a request to delete a specified document from a primary shard of a sharded database. Then, at block 1104, the process inserts a logical delete document identifying the specified document into the logical delete shard. The specified document remains in the primary shard. Then, at block 1106, the process receives a first query from a client application. The specified document satisfies the first query. Then, at block 1108, the process prevents the specified document from being returned in response to the first query while the logical delete document associated with the specified document remains in the logical delete shard.

[0088] 12A and 12B, these figures illustrate a flowchart of an exemplary process 1200 for logical deletion of data in a sharded database, according to an exemplary embodiment. In some embodiments, database 302, sharded database 400, sharded database 500, sharded database 600, sharded database 700, sharded database 800, or sharded database 900 perform process 1200.

[0089] In one embodiment, the process checks for an incoming database command at block 1202. Then, at block 1204, if the process receives a delete command, the process proceeds to block 1206; if the process receives a restore command, the process proceeds to block 1210; if the process receives a query command, the process proceeds to block 1212; and if the process does not receive a command, the process proceeds to block 1226.

[0090] If the process receives a delete command, then in block 1206, the process creates a new logical delete document in the logical delete shard that identifies the document specified by the delete command. Then, in block 1209, the process updates the logical delete shard sequence to reflect the newly added logical delete document.

[0091] If the process receives a restore command, the process removes the logically deleted document that identifies the document specified in the restore request from the logically deleted shard in block 1210. Then, in block 1208, the process updates the logically deleted shard sequence to reflect the newly removed logically deleted document.

[0092] If the process receives a query command, then at block 1212, the process executes the query in the primary shard and aggregates the query results. Next, at block 1214, the process determines whether a completed index is available for the logically deleted shard. If such an index is available, the process proceeds to block 1216; if not, the process proceeds to block 1218. If the process determines at block 1216 that a completed index is available for the SDS, then the process executes the query in the SDS using the SDS index. On the other hand, if the process determines at block 1218 that a completed index is not available for the SDS, then the process executes the query in the SDS using a full table scan. At block 1220, the process determines whether the query at block 1216 or block 1218 returned results. If it did, the process proceeds to block 1222; if it did not, the process proceeds to block 1224. At block 1222, because the same query is performed on the primary shard and the SDS, and because documents in the SDS are essentially copies of documents from the logically deleted primary shard, if the query aligns with documents from the SDS, the query will also align with documents from the logically deleted primary shard. Thus, at block 1222, the process removes from the aggregated primary shard query results any query results identified by the SDS documents returned as query results. This action removes the logically deleted documents from the search results, presumably because deleted documents are not expected to be included in the query results. Then, at block 1224, the process responds to the query by returning the aggregated primary shard query results (which do not include results removed by SDS reconciliation).

[0093] If the process has not received a command, then at block 1226, the process checks whether the retention period for any SDS documents has expired. If so, the process continues by executing blocks 1228, 1230, 1232, and 1234 for each SDS document whose retention period has expired. In one embodiment, executing blocks 1228, 1230, 1232, and 1234 is equivalent to hard deleting the document from the sharded database. At block 1228, the process deletes the document identified by the expired SDS document from the primary shard. Next, at block 1230, the process updates the index for the primary shard to reflect the deletion of the document from the primary shard. Next, at block 1232, the process updates the SDS index to reflect the deletion of the expired SDS document. Finally, at block 1234, the process deletes the expired SDS document.

[0094] The following definitions and abbreviations are used for the purposes of interpreting the claims and this specification. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," or "containing," or other variations of these terms, are intended to cover non-exclusive inclusion. For example, a composition, mixture, process, method, article, or device that includes a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent in such composition, mixture, process, method, article, or device.

[0095] Moreover, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" are understood to include any integer number greater than or equal to one, i.e., 1, 2, 3, 4, etc. The term "plurality" is understood to include any integer number greater than or equal to two, i.e., 2, 3, 4, 5, etc. The term "connected" can include an indirect "connected" and a direct "connected."

[0096] References herein to "one embodiment," "embodiment," "exemplary embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but that all embodiments may or may not include that particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described with respect to one embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic with respect to other embodiments, whether or not explicitly stated.

[0097] The terms "about," "substantially," "approximately," and variations of these terms are intended to include the degree of error associated with the magnitude of the particular quantity based on the equipment available at the time of filing this application. For example, "about" can include a range of ±8%, 5%, or 2% of a given value.

[0098] The foregoing description of various embodiments of the present invention has been provided for illustrative purposes and is not intended to be exhaustive or to limit the foregoing to the disclosed embodiments. Many modifications and variations that do not depart from the scope of the described embodiments will be apparent to those skilled in the art. The terminology used herein has been selected to best explain the principles, practical applications, or technical improvements of the embodiments over commercially available technology, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0099] Thus, in these exemplary embodiments, a computer-implemented method, system or apparatus, and computer program product are provided for managing participation in an online community and other related features, functions, or operations. Where an embodiment or portions thereof are described with respect to one type of device, the computer-implemented method, system or apparatus, computer program product, or portions thereof, is adapted or configured for use with an appropriate comparable representation of that type of device.

[0100] Where embodiments are described as being implemented in an application, it is contemplated within the scope of exemplary embodiments to deliver the application in a Software-as-a-Service (SaaS) model. In a SaaS model, the functionality of an application implementing an embodiment is provided to users by running the application within a cloud infrastructure. Users can access the application using various client devices through a thin client interface (e.g., web-based email) such as a web browser or other lightweight client application. Users do not manage or control the underlying cloud infrastructure, including the cloud infrastructure's network, servers, operating system, or storage. In some cases, users may not manage or control the functionality of a SaaS application. In other cases, a SaaS implementation of an application may allow for possible exceptions to restricted user-specific application configuration settings.

[0101] The present invention may be a system, method, or computer program product, or combination thereof, at any level of technical detail that allows integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0102] The computer-readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves having instructions recorded thereon, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as being a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating in a waveguide or other transmission body (e.g., a light pulse traveling in a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0103] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each corresponding computing / processing device, or can be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. This network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each corresponding computing / processing device.

[0104] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or configuration data for an integrated circuit, or may be source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or the like. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or remote server. In the last scenario above, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to carry out aspects of the present invention.

[0105] Aspects of the present invention are described herein with reference to flowchart and / or block diagram illustrations of methods, apparatus (systems) and computer program products according to embodiments of the invention, it being understood that each block of those flowchart and / or block diagram illustrations, and combinations of blocks in those flowchart and / or block diagram illustrations, can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions may be provided to a general-purpose computer processor, a special-purpose computer processor, or other programmable data processing apparatus processor in such a manner that the instructions, executed by the computer processor or other programmable data processing apparatus processor, produce means for performing the functions / acts specified in the blocks of the flowchart and / or block diagrams to produce a machine. These computer-readable program instructions may also be stored on a computer-readable storage medium in such a manner that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions / operations specified in the blocks of the flowchart and / or block diagrams, and may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0107] These computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process in such a manner that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in the blocks of the flowchart and / or block diagrams.

[0108] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions that implement the specified logical function(s). In some alternative implementations, the functions shown in the blocks may be performed in an order different from that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0109] Embodiments of the present invention can also be delivered as part of a service agreement with a client company, a nonprofit organization, a government entity, an internal organizational structure, or the like. Aspects of these embodiments can include configuring a computer system to execute and deploy software, hardware, and web services that implement some or all of the methods described herein. Aspects of these embodiments can further include analyzing client operations, generating recommendations in response to the analysis, building a system that implements portions of the recommendations, integrating the system into existing processes and infrastructure, metering system usage, allocating costs to users of the system, and charging for system usage. While each of the above embodiments of the present invention has been described by describing their individual advantages, the present invention is not limited to any particular combination thereof. Conversely, such embodiments can be combined in any manner and number in accordance with the intended deployment of the present invention without losing their beneficial effects.

Claims

1. 1. A computer-implemented method comprising: Receiving a request to remove a specified document from a primary shard of a sharded database; inserting a logically deleted document identifying the specified document into a logically deleted shard, the specified document remaining in the primary shard; receiving a query from a client application, the specified documents satisfying the query; and Preventing the specified document from being returned in response to the query while the logically deleted document associated with the specified document remains in the logically deleted shard.

20. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the sharded database is a NoSQL database.

3. receiving a restore request to restore the specified document into the sharded database; and Restoring the specified document to the sharded database in response to the restore request, wherein the restoring includes removing the logically deleted document from the logically deleted shard. The computer-implemented method of claim 1 further comprising:

4. 2. The computer-implemented method of claim 1, wherein the logically deleted document includes a unique identifier (UID) of the specified document and a pointer to the specified document.

5. Executing the query on the primary data shard. The computer-implemented method of claim 4 further comprising:

6. executing the query against the logically deleted shard; and identifying logically deleted search results by detecting that the logically deleted documents returned from the query on the logically deleted shard are consistent with the specified documents returned from the query on the primary data shard. The computer-implemented method of claim 5 further comprising:

7. identifying the logically deleted search results, detecting that the UID of the logically deleted document returned from the query on the logically deleted shard matches the UID of the specified document returned from the query on the primary data shard.

7. The computer-implemented method of claim 6.

8. removing the specified document from being returned from the query on the primary data shard in response to identifying the logically deleted search result. The computer-implemented method of claim 6 further comprising:

9. Responsive to the insertion of the logically deleted document into the logically deleted shard, building a logical delete index of the logically deleted shard for the logically deleted document. The computer-implemented method of claim 5 further comprising:

10. determining, in response to receiving the query, that the logical delete index is incomplete; and responsive to determining that the logical delete index is incomplete, executing the query on the logical delete shard using a full table scan. The computer-implemented method of claim 9 further comprising:

11. determining, in response to receiving the query, that the logical delete index is complete; and responsive to determining that the logical delete index is complete, executing the query on the logical delete shard using the logical delete index. The computer-implemented method of claim 10 further comprising:

12. Detecting that the time since receiving the request to delete the specified document has reached a specified retention period; and running a physical delete process that purges the specified documents from the sharded database. The computer-implemented method of claim 1 further comprising:

13. The physical deletion process Deleting the specified document from the primary shard of the sharded database; updating the index of the primary shard to reflect the removal of the specified document from the primary shard; updating a logical delete index of the logical delete shard to reflect the deletion of the logically deleted document from the logical delete shard; and Removing the logically deleted document from the logically deleted shard. The computer-implemented method of claim 12, comprising:

14. A computer program having program instructions, comprising: Receiving a request to remove a specified document from a primary shard of a sharded database; inserting a logically deleted document identifying the specified document into a logically deleted shard, the specified document remaining in the primary shard; receiving a query from a client application, the specified documents satisfying the query; and Preventing the specified document from being returned in response to the query while the logically deleted document associated with the specified document remains in the logically deleted shard. A computer program that causes a processor to perform operations including:

15. 15. The computer program product of claim 14, wherein the program instructions are stored on a computer-readable storage device within a data processing system, and wherein the stored program instructions are transferred across a network from a remote data processing system.

16. the program instructions are stored in a computer readable storage device within a server data processing system, and the stored program instructions are downloaded, in response to a request, across a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system; the processor, program instructions for metering usage of said program instructions associated with said request; generating an invoice based on the metered usage; and 15. The computer program product of claim 14, 17. The processor, receiving a restore request to restore the specified document into the sharded database; and Restoring the specified document to the sharded database in response to the restore request, wherein the restoring includes removing the logically deleted document from the logically deleted shard.

15. The computer program product of claim 14, further comprising:

18. The processor, executing the query on a primary data shard; executing the query against the logically deleted shard; and identifying logically deleted search results by detecting that the logically deleted documents returned from the query on the logically deleted shard are consistent with the specified documents returned from the query on the primary data shard.

15. The computer program product of claim 14, further comprising:

19. 1. A computer system, the computer system including a processor and one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions being operable by the processor to: Receiving a request to remove a specified document from a primary shard of a sharded database; inserting a logically deleted document identifying the specified document into a logically deleted shard, the specified document remaining in the primary shard; receiving a query from a client application, the specified documents satisfying the query; and Preventing the specified document from being returned in response to the query while the logically deleted document associated with the specified document remains in the logically deleted shard. a computer system executable to cause the processor to perform operations including:

20. receiving a restore request to restore the specified document into the sharded database; and Restoring the specified document to the sharded database in response to the restore request, wherein the restoring includes removing the logically deleted document from the logically deleted shard.

20. The computer system of claim 19, further comprising:

Images