Soft deletion of data in a sharded database

Abstract

A method, system, and computer program product for a sharded database are provided. The method includes receiving a request to delete a specified document from a primary shard of a sharded database. The method also includes inserting a soft delete document into a soft delete shard, where the soft delete document identifies the specified document, the specified document remaining in the primary shard. The method also includes receiving a query from a client application, where the specified document satisfies the query. The method also includes preventing the specified document from being returned in response to the query while the soft delete document associated with the specified document remains in the soft delete shard.

Description

Technical Field

[0001] This invention generally relates to methods, systems, and computer program products for sharded databases. More specifically, this invention relates to methods, systems, and computer program products for soft-deleting data in a sharded database. Background Technology

[0002] Modern database systems offer the ability to store, search, and retrieve information quickly. However, the volume of digital content is growing exponentially, requiring massive storage systems to store and manage it. Therefore, modern databases often interact with or are part of computer applications that collect, update, analyze, or report large datasets.

[0003] When datasets are so large and access demands are so high that they reach the performance or storage limits of a single server, the data can be distributed across multiple servers to provide additional performance and storage capacity. Each segment in such a distributed database system is called a "shard." The functionality of each shard can be allocated according to strategies designed to distribute user load and optimize the performance of the database system. Allocating or "sharding" the database in this way can overcome performance and storage limitations. Summary of the Invention

[0004] The illustrative embodiments provide soft deletion of data in a sharded database. The embodiments include receiving a request to delete a specified document from the primary shard of the sharded database. The embodiments also include inserting a soft-deleted document into the soft-deleted shard, wherein the soft-deleted document identifies the specified document retained in the primary shard. The embodiments also include receiving a query from a client application, wherein the specified document satisfies the query. The embodiments also include preventing the specified document from being returned in response to the query, while the soft-deleted document associated with the specified document remains in the soft-deleted shard. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the embodiments. This embodiment allows for soft deletion and recovery of primary data without any write or delete operations on the primary shard and without any changes to the indexes of the primary shard, thereby eliminating such time-consuming processes and resulting in a significant performance improvement over previous soft deletion procedures for NoSQL databases.

[0005] In another embodiment, the embodiment further includes receiving a recovery request to restore a specified document to the sharded database, and restoring the specified document to the sharded database in response to the recovery request, wherein the recovery includes removing the soft-deleted document from the soft-deleted shard. Therefore, the embodiment allows the recovery of soft-deleted data in response to a recovery request, thereby providing the intended functionality to end users already accustomed to having the ability to undo or restore data and avoiding costly data loss.

[0006] In another embodiment, the embodiment further includes detecting that the amount of time elapsed since receiving the request to delete the specified document has reached a predetermined retention period, and performing a hard delete process to remove the specified document from the sharded database. Therefore, the embodiment allows a period of time for data recovery and permanently removes the data after that period to prevent unwanted data from consuming database resources.

[0007] The embodiments include a computer-usable program product. The computer-usable program product includes a computer-readable storage medium and program instructions stored on the storage medium.

[0008] The embodiments include a computer system. The computer system includes a processor, a computer-readable storage medium, a computer-readable storage medium, and program instructions stored on the storage medium for execution by the processor via the memory. Attached Figure Description

[0009] The appended claims set forth novel features that are considered characteristic of the invention. However, the invention itself, its preferred modes of use, further objects and advantages, can be best understood by referring to the following detailed description of illustrative embodiments taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 A block diagram depicts a network of a data processing system that can implement illustrative embodiments;

[0011] Figure 2 A block diagram of a data processing system that can implement illustrative embodiments is depicted;

[0012] Figure 3 A block diagram of an example fragmentation system according to an illustrative embodiment is depicted;

[0013] Figure 4 A block diagram of a sharded database according to an illustrative embodiment is depicted;

[0014] Figure 5 A block diagram of a sharded database according to an illustrative embodiment is depicted;

[0015] Figure 6 A block diagram of an example sharded database according to an illustrative embodiment is depicted;

[0016] Figure 7 A block diagram of an example sharded database according to an illustrative embodiment is depicted;

[0017] Figure 8 A block diagram of an example sharded database according to an illustrative embodiment is depicted;

[0018] Figure 9A block diagram of an example sharded database according to an illustrative embodiment is depicted;

[0019] Figure 10 A block diagram depicting the timeline process of the SDS index building process according to an illustrative embodiment;

[0020] Figure 11 A flowchart depicts an example process for softly deleting data in a sharded database according to an illustrative embodiment;

[0021] Figure 12A A first portion of a flowchart depicts an example process for soft-deleting data in a sharded database according to an illustrative embodiment; and

[0022] Figure 12B The second part of the flowchart depicts an example process for softly deleting data in a sharded database according to an illustrative embodiment. Detailed Implementation

[0023] Modern databases include cross-platform, document-oriented NoSQL (not just Structured Query Language) databases. These databases avoid the traditional table-based relational database structure to facilitate sharding. When a rapidly growing database runs on a single server, it will eventually reach the limits of the server's available computing resources. These limits include capacity constraints for storing data and processing limitations for handling queries and other database commands.

[0024] Sharding involves dividing data into two or more subsets stored on separate servers. Such subsets are referred to as “primary shards” or “major data shards” in this paper. This distributed architecture can support deployments with very large datasets and high-throughput operations. Therefore, sharding facilitates scalable setups for storing large amounts of data across multiple primary shards on corresponding servers.

[0025] Sharded databases can also include a data replication mechanism, where the database creates a set of copies of each shard that maintains the same data. At any given time in the replica set, only one shard acts as the primary shard, and all other replica shards are secondary shards. All write and read operations are routed to the primary shard and then evenly distributed (if necessary) to the other secondary shards in the set.

[0026] While sharding offers advantages in terms of scalability, the problem lies in the fact that adding or deleting data from the primary shard is a relatively time-consuming process, negatively impacting database performance. When data is added to or deleted from the primary shard, the primary index must be rebuilt to update the index records to reflect the change. Indexes are used to access the database, and large databases may require several large indexes that need to be maintained for efficient database access. When the database changes, indexes typically need to be maintained or updated. This re-indexing can take a long time and render the indexes unusable for queries until the update is complete.

[0027] Past attempts to implement soft delete processes in sharded databases have complicated this problem. When data is soft deleted, it becomes unavailable for selection or use, but can be recovered using the database or server's normal and available functions. Conversely, without soft delete, data is always hard deleted, meaning it is permanently lost and cannot be recovered without additional effort (which may or may not be effective).

[0028] Data loss can be extremely costly and frustrating for end users accustomed to the ability to undo or recover data. Therefore, implementing soft delete functionality is crucial for preventing data loss and providing users with the expected functionality. However, implementing soft delete is a system-specific practice that presents unique challenges for some types of systems while posing no problem for others. For example, past efforts to implement soft delete in document-oriented NoSQL databases negatively impacted database performance. This was because past soft delete processes involved writing a flag or other such data to the primary shard to mark the "soft-deleted" data as unavailable, and when the "soft-deleted" data was cleared, both the "soft-deleted" data and the flag were deleted; additionally, for each of these write and delete operations, an index update operation was performed on the shard index. However, performing write, delete, and index update operations on the database is expensive in terms of time and processing. Therefore, these past types of soft delete techniques present a technical problem: they require several additional, expensive operations, resulting in an undesirable performance trade-off for the benefit of recoverable data.

[0029] To address these technical challenges, the disclosed embodiments introduce an additional shard (referred to as a soft-delete shard (SDS)) that stores soft-deleted documents (SDDs) that reference the soft-deleted primary data. Unlike secondary shards, which maintain copies of data from the primary shard regardless of the primary shard's state, the SDS stores only documents associated with primary shard documents that have been soft-deleted but not yet cleared. During this period, the referenced primary document remains in the primary shard unless and until it is cleared, i.e., hard-deleted. If soft-deleted data is recovered instead of cleared, recovery involves removing the associated SDD from the SDS. This allows for soft deletion and recovery of primary data without any write or delete operations on the primary shard and without any changes to the primary shard's indexes. Eliminating these time-consuming processes results in a significant performance improvement over previous soft-delete procedures used for NoSQL databases.

[0030] An illustrative embodiment of a sharded database application with soft deletion receives a request to delete a specified document from the primary shard of the sharded database, and in response, inserts an SDD identifying the specified document into the SDS, leaving the specified document in the primary shard. Subsequently, if the application receives a query from a client application that satisfies the query for the specified document, the application prevents the specified document from being returned along with the query results, as long as the SDD associated with the specified document remains in the SDS.

[0031] In illustrative embodiments, database applications perform operations on a NoSQL database. Embodiments include any of a wide range of technologies and architectures (including NoSQL). For example, in some embodiments, a NoSQL database is a document-oriented database or document store that stores data in the form of documents, each document having a unique identifier (UID), i.e., some metadata that provides a degree of structure to the data, which can be formatted according to any of a variety of known data formats. And document data is stored in any desired format. In some embodiments, multiple database servers collectively provide services to the NoSQL database. Thus, in some embodiments, a NoSQL database includes a non-relational and primarily distributed database system that allows for rapid, specialized organization and analysis of extremely high volumes of entirely different data types. In some embodiments, a NoSQL database includes databases referred to as cloud databases, non-relational databases, big data databases, and / or numerous other terms used for NoSQL databases.

[0032] Implementations of an SDD can reference a specified document in any of a variety of different ways. For example, in some embodiments, the SDD identifies the specified document by including a unique identifier (UID) for that document. Alternatively or additionally, in some embodiments, the SDD includes a pointer to the specified document.

[0033] In some embodiments, if an application receives a query from a client application, the database application executes the query against the primary data shard and against the SDS. The application identifies that the query results from the primary shard will include soft-deleted documents by detecting the SDD returned from the query against the SDS. The application identifies soft-deleted documents by evaluating the SDD. Each SDD uniquely identifies a soft-deleted document from the primary shard, so the application uses this information to locate soft-deleted documents from the query results returned from the primary shard. For example, in some embodiments, the SDD includes the UID of the soft-deleted document in the primary shard, and the application locates the soft-deleted document from the query results by locating a document in the query results that has the UID of the SDD returned from the query against the SDS. After the application locates the soft-deleted document in the query results, the application removes the soft-deleted document from the query results. Since the query results do not include soft-deleted documents from the primary shard, the query results are presented as expected by excluding soft-deleted documents.

[0034] In illustrative embodiments, an application builds one or more SDS indexes to facilitate querying of the SDS. In some embodiments, the index includes associating a key with the location of the corresponding SDS. In some embodiments, the database application builds one or more indexes on the primary shard.

[0035] Depending on specific implementation considerations, many different types of indexes exist that can be used by an application. For example, for unstructured data or human language data, a full-text index can be used, which has a language analyzer that transforms text chunks into index entries. For geospatial or geotemporal data, points, polygons, and other shapes can be indexed in multidimensional space. In some embodiments, an application builds one or more SDS indexes to match the number and type of indexes that exist for the primary shard. This allows queries on the SDS to be consistent with queries on the primary shard.

[0036] In the illustrative embodiment, when an application executes a query against an SDS, the application first checks the status of the SDS index to determine whether the SDS index is complete or incomplete. If the application determines that the SDS index is complete, the application uses the SDS index to execute the SDS query. Otherwise, if the application determines that the SDS index is incomplete, the application uses a full table scan to execute the SDS query.

[0037] In illustrative embodiments, after a document in the primary shard is soft-deleted, it remains soft-deleted and can be recovered within a specified retention period. In some embodiments, this specified retention period is a time period set by the user. The application periodically checks if the amount of time a soft-deleted document has been soft-deleted is equal to or greater than the specified retention period. After the specified retention period has elapsed, the application removes the soft-deleted document from the primary shard. In some embodiments, the application removes soft-deleted documents from the shard database by performing a hard delete procedure on the specified document. In some such embodiments, the hard delete procedure includes deleting the specified document from the primary shard of the shard database, subsequently updating the primary shard's index, subsequently updating the soft delete index of the SDS, and subsequently deleting the SDD identifying the specified document.

[0038] For clarity of description and without implying any limitation thereof, some example configurations are used to describe illustrative embodiments. Based on this disclosure, those skilled in the art will be able to conceive of many variations, adaptations, and modifications to the construction for achieving the described objectives, and these are all considered to be within the scope of the exemplary embodiments.

[0039] Furthermore, simplified diagrams of the data processing environment are used in the accompanying drawings and illustrative embodiments. In a real computing environment, there may be additional structures or components not shown or described herein, or structures or components that differ from those shown but are used for functions similar to those described herein, without departing from the scope of the illustrative embodiments.

[0040] Furthermore, illustrative embodiments are described only as examples, relating to specific actual or hypothetical components. The steps described by the various illustrative embodiments can be adapted to provide an explanation of decisions made by, for example, a machine learning classifier model.

[0041] Any particular manifestation of these and other similar human factors is not intended to limit the invention. Any suitable manifestation of these and other similar products may be chosen within the scope of the exemplary embodiments.

[0042] The examples in this disclosure are for illustrative purposes only and are not intended to limit the scope of the illustrative embodiments. Any advantages listed herein are merely examples and are not intended to limit the illustrative embodiments. Additional or different advantages may be achieved through specific illustrative embodiments. Furthermore, specific illustrative embodiments may have some, all, or none of the advantages listed above.

[0043] Furthermore, illustrative embodiments can be implemented for any type of data, data source, or access to a data source via a data network. Within the scope of this invention, any type of data storage device can provide data locally at a data processing system or via a data network to embodiments of the invention. When embodiments are described using mobile devices, within the scope of the illustrative embodiments, any type of data storage device suitable for use with mobile devices can provide data locally at the mobile device or via a data network to such embodiments.

[0044] The illustrative embodiments are described using specific code, comparative explanations, computer-readable storage media, advanced features, historical data, designs, architectures, protocols, layouts, diagrams, and tools, which are merely examples and not intended to limit the illustrative embodiments. Furthermore, for clarity, specific software, tools, and data processing environments are used in some instances as examples to describe the illustrative embodiments. The illustrative embodiments can be used in conjunction with other equivalent or similar purpose structures, systems, applications, or architectures. For example, within the scope of this invention, other equivalent mobile devices, structures, systems, applications, or their architectures can be used in conjunction with such embodiments of the invention. The illustrative embodiments can be implemented in hardware, software, or a combination thereof.

[0045] The examples in this disclosure are for illustrative purposes only and are not intended to limit the scope of the illustrative embodiments. Additional data, operations, actions, tasks, activities, and manipulations may be conceived from this disclosure, and such additional data, operations, actions, tasks, activities, and manipulations are contemplated within the scope of the illustrative embodiments.

[0046] Any advantages listed herein are merely illustrative and are not intended to limit the illustrative embodiments. Additional or different advantages may be achieved through specific illustrative embodiments. Furthermore, specific illustrative embodiments may have some, all, or none of the advantages listed above.

[0047] Refer to the attached diagram, and especially refer to... Figure 1 and 2 These figures are example diagrams of a data processing environment in which illustrative embodiments can be implemented. Figure 1 and Figure 2 This is merely an example and is not intended to assert or imply any limitation regarding the environment in which different embodiments may be implemented. Specific implementations may make many modifications to the depicted environment based on the following description.

[0048] Figure 1A block diagram depicts a network in which an illustrative embodiment of a data processing system may be implemented. Data processing environment 100 is a computer network in which the illustrative embodiment may be implemented. Data processing environment 100 includes network 102. Network 102 is a medium for providing communication links between various devices and computers connected together within data processing environment 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0049] The client or server is merely an example role for certain data processing systems connected to network 102 and is not intended to exclude other configurations or roles of these data processing systems. Data processing system 104 is coupled to network 102. Software applications can execute on any data processing system in data processing environment 100. (Described as...) Figure 1 Any software application executing in processing system 104 can be configured to execute in another data processing system in a similar manner. Figure 1 Any data or information stored or generated in the data processing system 104 may be configured to be stored or generated in another data processing system in a similar manner. A data processing system such as data processing system 104 may contain data and may have software applications or software tools that perform computational processing thereon. In an embodiment, data processing system 104 includes a memory 124 that includes an application 105A configured to implement one or more data processor functions according to one or more embodiments described herein.

[0050] Server 106 is coupled to network 102 along with storage unit 108. Storage unit 108 includes database 109, which is configured to store data as described herein with respect to various embodiments, such as image data and attribute data. Server 106 is a conventional data processing system. In embodiments, server 106 includes processing elements of stream processing application 105B, which may be configured to implement one or more of the processor functions described herein according to one or more embodiments.

[0051] Clients 110, 112, and 114 are also coupled to network 102. A conventional data processing system, such as server 106 or clients 110, 112, or 114, may contain data and may have software applications or software tools that perform conventional computational processes on it.

[0052] This is merely an example and does not imply any limitations on such an architecture. Figure 1Certain components that may be used in example implementations of the embodiments are depicted. For example, server 106 and clients 110, 112, and 114 are depicted as servers, while clients are merely examples and do not imply any limitation on the client-server architecture. As another example, embodiments may be distributed across several data processing systems and a data network as shown in the figures, while another embodiment may be implemented on a single data processing system within the scope of the illustrative embodiments. Conventional data processing systems 106, 110, 112, and 114 also represent example nodes suitable for implementing the embodiments in clusters, partitions, and other configurations.

[0053] Device 132 is an example of a conventional computing device described herein. For example, device 132 may take the form of a smartphone, tablet computer, laptop computer, client 110 in fixed or portable form, wearable computing device, or any other suitable device. In an embodiment, device 132 sends a request to server 106 to perform one or more data processing tasks by stream processing application 105B, such as initiating the processes described herein. Figure 1 Any software application running in another conventional data processing system can be configured to run in device 132 in a similar manner. Figure 1 Any data or information stored or generated in another conventional data processing system can be configured to be stored or generated in device 132 in a similar manner.

[0054] Server 106, storage unit 108, data processing system 104, clients 110, 112, and 114, and device 132 can be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connections. Clients 110, 112, and 114 can be, for example, personal computers or network computers.

[0055] In the described example, server 106 can provide data, such as boot files, operating system images, and applications, to clients 110, 112, and 114. In this example, clients 110, 112, and 114 can be clients of server 106. Clients 110, 112, 114, or some combination thereof, can include their own data, boot files, operating system images, and applications. Data processing environment 100 may include additional servers, clients, and other devices not shown.

[0056] In the depicted example, memory 124 can provide data, such as boot files, operating system images, and applications, to processor 122. Processor 122 may include its own data, boot files, operating system images, and applications. Data processing environment 100 may include additional memory, processors, and other devices not shown.

[0057] In the described example, data processing environment 100 can be the Internet. Network 102 can represent a collection of networks and gateways that communicate with each other using Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols. At the core of the Internet is the backbone of data communication links between master nodes or host computers, including thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, data processing environment 100 can also be implemented as many different types of networks, such as intranets, local area networks (LANs), or wide area networks (WANs). Figure 1 This is intended as an example, not as an architectural limitation on different illustrative embodiments.

[0058] Among other uses, the data processing environment 100 can be used to implement a client-server environment in which exemplary embodiments can be implemented. The client-server environment enables software applications and data to be distributed across a network, allowing applications to function through interoperability between conventional client-side and conventional server-side data processing systems. The data processing environment 100 can also employ a service-oriented architecture, where interoperable software components distributed across the network can be packaged together as a consistent business application. The data processing environment 100 can also take the form of a cloud and employ a service-delivered cloud computing model to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with service providers.

[0059] refer to Figure 2 The figure depicts a block diagram of a data processing system that can implement illustrative embodiments. Data processing system 200 is an example of a conventional computer, such as... Figure 1 The data processing system 104, server 106, or client 110, 112, and 114, or another type of device, may contain computer-available program code or instructions that implement these processes.

[0060] Data processing system 200 also represents a conventional data processing system or a configuration thereof, such as Figure 1 The data processing system 132 is a conventional data processing system in which computer-usable program code or instructions for implementing the processes of exemplary embodiments can be placed. The data processing system 200 is described as a computer by way of example only and is not limited thereto. Without departing from the general description of the operation and function of the data processing system 200 described herein, other devices (such as...) are also included. Figure 1 The implementation of the device (132) in the data processing system 200 can be modified, for example, by adding a touch interface, and even by removing certain depicted components from the data processing system 200.

[0061] In the described example, the data processing system 200 employs a hub architecture, including a Northbridge and Memory Controller Hub (NB / MCH) 202 and a Southbridge and Input / Output (I / O) Controller Hub (SB / ICH) 204. A processing unit 206, main memory 208, and a graphics processor 210 are coupled to the Northbridge and Memory Controller Hub (NB / MCH) 202. The processing unit 206 may contain one or more processors and may be implemented using one or more heterogeneous processor systems. The processing unit 206 may be a multi-core processor. In some implementations, the graphics processor 210 may be coupled to the NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0062] In the depicted example, a local area network (LAN) adapter 212 is coupled to the Southbridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a universal serial bus (USB) and other ports 232, and a PCI / PCIe device 234 are coupled to the Southbridge and I / O controller hub 204 via bus 238. A hard disk drive (HDD) or solid-state drive (SSD) 226 and a CD-ROM 230 are coupled to the Southbridge and I / O controller hub 204 via bus 240. The PCI / PCIe device 234 may include, for example, an Ethernet adapter, an add-in card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. The ROM 224 may be, for example, a flash binary input / output system (BIOS). Hard disk drive 226 and CD-ROM 230 may use, for example, integrated drive electronics (IDE), Serial Advanced Technology Attachment (SATA) interface, or variants such as external SATA (eSATA) and micro SATA (mSATA). Super I / O (SIO) device 236 may be coupled to the southbridge and I / O controller hub (SB / ICH) 204 via bus 238.

[0063] Memory (such as main memory 208, ROM 224, or flash memory (not shown)) is some example of computer-available storage devices. Hard disk drives or solid-state drives 226, CD-ROMs 230, and other similar available devices are some examples of computer-available storage devices that include computer-available storage media.

[0064] The operating system runs on the processing unit 206. The operating system coordinates and provides services to... Figure 2The data processing system 200 controls various components within it. The operating system can be a commercially available operating system for any type of computing platform, including but not limited to server systems, personal computers, and mobile devices. Object-oriented or other types of programming systems can operate alongside the operating system and provide calls to the operating system from programs or applications executing on the data processing system 200.

[0065] For use in operating systems, object-oriented programming systems, and applications or programs (such as...) Figure 1 The instructions for application 105 are located on a storage device, such as in the form of code 226A on hard disk drive 226, and may be loaded into at least one of one or more memories (such as main memory 208) for execution by processing unit 206. The processes of the illustrative embodiment may be executed by processing unit 206 using computer-implemented instructions that may reside in memory, such as main memory 208, read-only memory 224, or in one or more peripheral devices.

[0066] Furthermore, in one scenario, code 226A can be downloaded from remote system 201B via network 201A, where similar code 201C is stored on storage device 201D. In another scenario, code 226A can be downloaded to remote system 201B via network 201A, where the downloaded code 201C is stored on storage device 201D.

[0067] Figure 1-2 The hardware within can vary depending on the implementation. Besides... Figure 1-2 In addition to the hardware described herein, or as an alternative, other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives, may be used.

[0068] In some illustrative examples, the data processing system 200 may be a personal digital assistant (PDA), which is typically configured with flash memory to provide non-volatile memory for storing operating system files and / or user-generated data. The bus system may include one or more buses, such as a system bus, I / O bus, and PCI bus. Of course, the bus system can be implemented using any type of communication structure or architecture that provides data transfer between different components or devices attached to that structure or architecture.

[0069] The communication unit may include one or more devices for sending and receiving data, such as a modem or network adapter. Memory may be, for example, main memory 208 or cache, such as the cache found in the Northbridge and memory controller hub 202. The processing unit may include one or more processors or CPUs.

[0070] Figure 1-2The examples depicted and those described above are not intended to imply architectural limitations. For example, the data processing system 200 could take the form of a tablet computer, laptop computer, or telephone device, in addition to being a mobile or wearable device.

[0071] When a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates in a manner similar to the data processing system 200 using virtualized representations of some or all of the components depicted in the data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 is represented as a virtualized instance of all or some of the hardware processing units 206 available in the host data processing system, main memory 208 is represented as a virtualized instance of all or some portions of the main memory 208 available in the host data processing system, and disk 226 is represented as a virtualized instance of all or some portions of the disk 226 available in the host data processing system. In this case, the host data processing system is represented by data processing system 200.

[0072] refer to Figure 3 This figure depicts a block diagram of an example sharding system 300 according to an illustrative embodiment. In the illustrated embodiment, the sharding system 300 includes a sharded database 302, wherein one or more client applications 322 can access data items distributed across multiple primary database shards 312, 314, 316, and 318. In this embodiment, the sharded database 302 is... Figure 1 The example of database 109, and client application 322 is... Figure 1 Examples of applications of 105A / 105B.

[0073] Sharding system 300 includes client applications 322 and main database shards 312, 314, 316, and 318. The number of client applications 322 and main database shards 312, 314, 316, and 318 in system 300 can vary. In some embodiments, each of the main database shards 312, 314, 316, and 318 is a separate and independent database that does not need to know about any other shards within system 300. In some such embodiments, for example, each of the main database shards 312, 314, 316, and 318 includes a separate database server and relational database. In some embodiments, there are two or more client applications 322 operating on a separate computing system independent of another client 322. In some such embodiments, each of the clients 322 executes a separate instance of a software program utilizing a hash function to compute an identifier of the specific shard on which a specific data item is stored or will be stored in the main database shards 312, 314, 316, and 318, based on the master key of the specific data item. In some embodiments, such a data item is a separate record having different values ​​for a similar set of attributes.

[0074] Typically, to perform an operation related to a data item already stored on one of the main database shards 312, 314, 316, and 318, client application 322 executes one or more instructions related to the data item by issuing database commands to database 302. For example, the instructions may include query commands processed by one or more main query engines 308 or deletion commands processed by deletion manager 304. In some embodiments, database 302 includes main query engines 308 for each main database shard 312-318. Illustrative embodiments also allow recovery commands processed by recovery manager 324, which allow the recovery of deleted data within a specified time period.

[0075] According to an embodiment of the present invention, database 302 is capable of recovering deleted data because soft delete shard (SDS) 320 is included in system 300 and stores soft delete documents adapted to the soft delete state of documents deleted from main database shards 312, 314, 316, and 318. After a specified time period, as part of a hard delete process for clearing deleted data, cleanup manager 306 removes the soft delete documents from SDS 320. During the specified time period, SDS query engine 310 executes queries on SDS 320 corresponding to each query executed by main query engine(s) 308 on main database shards 312, 314, 316, and 318. If SDS query engine 310 finds a document in SDS 320, because the same query is performed on the sets of primary database shards 312, 314, 316, and 318 as well as SDS 320, and because the documents in SDS 320 are essentially copies or references to soft-deleted documents from primary database shards 312, 314, 316, and 318, this means that if the query matches a document from SDS 320, the same query will also match soft-deleted documents from primary database shards 312, 314, 316, and 318. Therefore, database 302 removes the query results identified by the SDS 320 document returned as query results from the primary shard query results. This action removes soft-deleted documents from the search results because it is not expected to include deleted documents in the query results.

[0076] refer to Figure 4 The figure depicts a block diagram of a sharded database 400 according to an illustrative embodiment. In this embodiment, the sharded database 400 is... Figure 3 Database 302 or Figure 1 Example of database 109.

[0077] In the illustrated embodiment, database 400 includes a deletion manager 402, a cleanup manager 404, a query subtractor 406, a query manager 408, main database shards 410, 412, and 414, an SDS 416, an index manager 418, index engines 420, 422, and 424, an index synchronizer 426, SDS indexes 428, 430, and 432, a query aggregator 434, main query engines 436, 438, and 440, an SDS query engine 442, and a recovery manager 444. In some embodiments, the functionality described herein is distributed across multiple systems, which may include a combination of software- and / or hardware-based systems (e.g., application-specific integrated circuits (ASICs), computer programs, or smartphone applications). In this embodiment, deletion manager 402 is an example of deletion manager 304, clear manager 404 is an example of clear manager 306, primary database shards 410, 412, and 414 are examples of primary database shards 312, 314, 316, and 318, SDS 416 is an example of SDS 320, primary query engines 436, 438, and 440 are examples of primary query engine 308, SDS query engine 442 is an example of SDS query engine 310, and recovery manager 444 is an example of recovery manager 324.

[0078] In the illustrated embodiment, the index manager 418 dispatches indexing requests to different primary shards 410, 412, and 414 by invoking the corresponding indexing engines 420, 422, and 424. Indexing engines 420, 422, and 424 execute the indexing requests. One indexing engine 420, 422, and 424 is shown for each primary shard 410, 412, and 414, while alternative embodiments include multiple indexing engines for each primary shard. In some such embodiments, the database 400 builds one or more indexes for primary shards 410, 412, and 414. Depending on implementation-specific considerations, many different types of indexes exist that can be used by an application. For example, for unstructured data or human language data, a full-text index with a language analyzer that transforms text chunks into index entries can be used. For geospatial or geotemporal data, points, polygons, and other shapes can be indexed in multidimensional space. In some such embodiments, the database 400 includes different types of indexing engines for the corresponding index types.

[0079] In some embodiments, index synchronizer 426 asynchronously builds and updates one or more SDS indexes 428, 430, and 432 for SDS 416 to match the number and type of existing indexes for each of the primary shards 410, 412, and 414. This allows queries to SDS 416 to be consistent with queries to primary shards 410, 412, and 414.

[0080] In some embodiments, if query manager 408 receives a query from a client application, query manager 408 instructs main query engines 436, 438, and 440 to execute queries against main data shards 410, 412, and 414, and also instructs SDS query engine 442 to execute a query against SDS 416. Query aggregator 434 receives query results from each of the main query engines 436, 438, and 440 and performs result aggregation to combine the results into an aggregated result set.

[0081] Query subtractor 406 receives any query results found by SDS query engine 442. Query subtractor 406 identifies soft-deleted documents in the aggregated result sets from primary shards 410, 412, and 414 by detecting that an SDD has been returned from a query against SDS 416. Query subtractor 406 is able to identify soft-deleted documents from the documents in the aggregated result sets by evaluating the SDD. Each SDD uniquely identifies a soft-deleted document from primary shards 410, 412, and 414, therefore query subtractor 406 uses this information to locate soft-deleted documents from the aggregated result sets.

[0082] For example, in some embodiments, the SDD includes the UID of soft-deleted documents in primary shards 410, 412, and 414, and the query subtractor 406 locates soft-deleted documents from the aggregated result set by locating documents in the query results that have the UID of the SDD returned from the query against SDS 416. After the query subtractor 406 locates a soft-deleted document in the query results, the query subtractor 406 removes the soft-deleted document from the query result set. Since the query result set does not include soft-deleted documents from primary shards 410, 412, and 414, the query result set is presented as expected by excluding the soft-deleted documents. The purge manager performs hard deletions on the master data and indexes in the primary and soft-deleted shards.

[0083] In illustrative embodiments, after documents in primary shards 410, 412, and 414 are soft-deleted, they remain soft-deleted and can be recovered by recovery manager 444 for a specified retention period. In some embodiments, this specified retention period is a time period set by the user. The application periodically checks for soft-deleted documents whose soft-deletion time is equal to or greater than the specified retention period. After a soft-deleted document has been soft-deleted for the specified retention period, cleanup manager 404 cleans up the soft-deleted documents from primary shards 410, 412, and 414. In some embodiments, cleanup manager 404 cleans up soft-deleted documents by performing a hard-delete procedure on specified documents from shard database 400. In some such embodiments, the hard delete process includes deleting the specified document from the primary shard 410, 412, or 414 of the sharded database 400, followed by updating the indexes by the index engines 420, 422, and 424 of the primary shards 410, 412, and 414, followed by updating the soft delete indexes 428, 430, and 432 of the SDS 416, and then deleting the SDD that identifies the specified document.

[0084] refer to Figure 5 The figure depicts a block diagram of a sharded database 500 according to an illustrative embodiment. In this embodiment, the sharded database 500 is... Figure 4 400 sharded databases Figure 3 Database 302 or Figure 1 Example of database 109.

[0085] In the illustrated embodiment, database 500 includes primary shards 502-516 and SDS 518. Each of primary shards 502-516 includes an equal number of addressable data stores. In the illustrated embodiment, each of primary shards 502-516 includes a 0x1F address; however, this amount can vary. Maintaining a balanced distribution of data across primary shards 502-516 is desirable. One reason for this is that if the shards become unbalanced (e.g., primary shard 502 is close to capacity while primary shard 504 is close to empty), RAM and disk space will be underutilized on one primary shard and overutilized on the other. Some embodiments attempt to store primary indexes along with recently used data in RAM to accelerate performance. In the case of imbalance, RAM used for overloaded shards begins to evict data items or even indexes from RAM. Therefore, maintaining a balanced data hierarchy across primary shards 502-516 is desirable.

[0086] On the other hand, SDS 518 stores data in a unique environment (soft-deleted data only). Therefore, in some embodiments, database 500 includes shard balancing to distribute data evenly across primary shards 502-516, but excludes SDS 518 from the balancing. Similarly, in some embodiments, shards 502-516 include primary and secondary data shards (e.g., where the secondary shard is used for data replication). In some such embodiments, shard balancing is performed to distribute data evenly across primary and secondary shards 502-516, but excludes SDS 518 from the shard balancing.

[0087] refer to Figure 6 The figure depicts a block diagram of an example sharded database 600 according to an illustrative embodiment. More specifically, Figure 6 The effect of a Document Deletion Command (DOCB) in a sharded database 600 is illustrated. In a particular embodiment, sharded database 600 is... Figure 5 500 sharded databases Figure 4 400 sharded databases Figure 3 Database 302 or Figure 1 Example of database 109.

[0088] In the illustrated embodiment, the sharded database 600 includes three primary shards: database shard A (DBSA) 602, database shard B (DBSB) 604, and database shard C (DBSC) 606. The sharded database 600 receives a delete command to delete DOCB from DBSB 604. This... Figure 6 The strikethrough of a document DOCB in DBSB 604 indicates that the DOCB has been soft-deleted as a result of a delete command. However, the strikethrough of a DOCB in DBSB 604 is merely symbolic and for illustrative purposes, as the actual document DOCB in DBSB 604 is not altered by delete or restore commands. Similarly, as a result of a delete command, the DOCB's SDD is added to SDS 608. The SDS index is then updated to reflect the DOCB's SDD added to SDS 608. When a DOCB is in a soft-deleted state, if query manager 610 receives a query that satisfies the DOCB, it returns the DOCB as a query result along with other query results. However, since the DOCB's SDD exists in SDS 608, the removal of the DOCB is still present in the pre-results. Therefore, final result 612 excludes the soft-deleted document DOCB.

[0089] refer to Figure 7 The figure depicts a block diagram of an example sharded database 700 according to an illustrative embodiment. More specifically, Figure 7The effect of a recovery command for a previously soft-deleted document (DOCB) is shown. In a particular embodiment, the sharded database 700 is... Figure 5 500 sharded databases Figure 4 400 sharded databases Figure 3 Database 302 or Figure 1 Example of database 109.

[0090] In the illustrated embodiment, the sharded database 700 includes three primary shards: database shard A (DBSA) 702, database shard B (DBSB) 704, and database shard C (DBSC) 706. In DBSB 704, the document DOCB is shown with a broken strikethrough, indicating that the DOCB was restored after being previously soft-deleted. However, the strikethrough of the DOCB in DBSB 704 is merely a symbol for illustrative purposes, as the delete or restore command does not change the actual document DOCB in DBSB 704. Instead, the actual change caused by the restore command is to SDS 708. As a result of the restore command, the SDS indexes of DOCB SDD and DOCB SDD are removed. Therefore, future queries to query manager 710 that return DOCB will include DOCB in both the preliminary and final results.

[0091] refer to Figure 8 The figure depicts a block diagram of an example sharded database 800 according to an illustrative embodiment. More specifically, Figure 8 The effect of a query command for soft deleted documents (DOCBs) in a sharded database 800 is shown. In a particular embodiment, sharded database 800 is... Figure 5 500 sharded databases Figure 4 400 sharded databases Figure 3 Database 302 or Figure 1 Example of database 109.

[0092] In the illustrated embodiment, the sharded database 800 includes three primary shards: database shard A (DBSA) 802, database shard B (DBSB) 804, and database shard C (DBSC) 806. In DBSB 804, a document DOCB is shown with a strikethrough, indicating that the DOCB has been soft-deleted. However, the strikethrough of the DOCB in DBSB 804 is merely an illustrative symbol, as delete or restore commands do not alter the actual document DOCB in DBSB 804. Therefore, SDS 808 includes the SDD of the DOCB, while the DOCB is in a soft-deleted state. When query manager 810 receives a query that satisfies the DOCB, the DOCB is returned as a query result along with other query results. However, because the SDD of the DOCB exists in SDS 808, the DOCB is removed but still present in the pre-results. Therefore, final result 812 excludes the soft-deleted document DOCB.

[0093] refer to Figure 9 The figure depicts a block diagram of an example sharded database 900 according to an illustrative embodiment. More specifically, Figure 9 This illustrates the effect of a hard deletion or cleanup action on a previously soft deleted document (DOCB). In a particular embodiment, the sharded database 900 is... Figure 5 500 sharded databases Figure 4 400 sharded databases Figure 3 Database 302 or Figure 1 Example of database 109.

[0094] In the illustrated embodiment, the sharded database 900 includes three primary shards: database shard A (DBSA) 902, database shard B (DBSB) 904, and database shard C (DBSC) 906. In DBSB 904, the document DOCB is shown with a strikethrough, indicating that the cleanup manager 910 permanently deletes the DOCB after a predetermined retention period since its soft deletion. Next, the cleanup manager 910 updates the primary shard's index to remove references to DOCB.FIELD5. Next, the cleanup manager 910 updates the soft-delete index of SDS 908. Finally, the cleanup manager 910 removes the SDD identifying the deleted document DOCB from SDS 908.

[0095] refer to Figure 10 This figure depicts a block diagram of the timeline process of the SDS index building process according to an illustrative embodiment. In a particular embodiment, Figure 10 The process shown is performed by index synchronizer 1010 (as... Figure 4 (Example of index synchronizer 426) is executed.

[0096] In the illustrative embodiment, the index synchronizer 1010 builds multiple SDS indexes 1012, 1014, and 1016 over a time period spanning from T1 to T4 to support efficient execution of SDS queries. If SDS indexes 1012, 1014, and 1016 are not completed (i.e., before T4), the database must perform a full table scan, that is, scan every document in the SDS to select SDDs that match the query statement. If SDS indexes 1012, 1014, and 1016 are completed (i.e., after T4), the database uses indexes 1012, 1014, and 1016 to limit the number of SDDs it will examine to select that match the query statement.

[0097] In some embodiments, index synchronizer 1010 builds multiple indexes for SDS, where each index is a different index type. In the illustrated embodiment, index synchronizer 1010 asynchronously builds indexes 1012, 1014, and 1016. Starting at time T1, from index state 1002, index synchronizer 1010 builds SDS index 1012, which completes at time T2 to index completion state 1004. At time T2, from index state 1004, index synchronizer 1010 builds SDS index 1014, which completes at time T3 to index completion state 1006. At time T3, from index state 1006, index synchronizer 1010 builds SDS index 1016, which completes at time T4 to index completion state 1008. In some embodiments, index synchronizer 1010 builds SDS indexes 1012, 1014, and 1016 to match the number and type of indexes existing for the primary shard. This allows queries on the SDS to be consistent with queries on the primary shard.

[0098] refer to Figure 11 The figure depicts a flowchart of an example process 1100 for softly deleting data in a sharded database according to an illustrative embodiment. In some embodiments, database 302, sharded database 400, sharded database 500, sharded database 600, sharded database 700, sharded database 800, or sharded database 900 perform process 1100.

[0099] In this embodiment, at block 1102, the process receives a request to delete a specified document from the primary shard of the sharded database. Next, at block 1104, the process inserts a soft-deleted document into the soft-delete shard, where the soft-deleted document identifies the specified document, which remains in the primary shard. Next, at block 1106, the process receives a first query from a client application, where the specified document satisfies the first query. Next, at block 1108, the process prevents the specified document from being returned in response to the first query, while the soft-deleted document associated with the specified document remains in the soft-delete shard.

[0100] refer to Figure 12A and12B These figures depict flowcharts of an example process 1200 for soft deletion of data in a sharded database according to an illustrative embodiment. In some embodiments, database 302, sharded database 400, sharded database 500, sharded database 600, sharded database 700, sharded database 800, or sharded database 900 perform process 1200.

[0101] In this embodiment, at block 1202, the process checks the incoming database command. Next, at block 1204, if the process receives a delete command, the process continues to block 1206; if the process receives a restore command, the process continues to block 1210; if the process receives a query command, the process continues to block 1212; if the process does not receive a command, the process continues to block 1226.

[0102] If the process receives a delete command, then in box 1206, the process creates a new soft delete document in the soft delete segment that identifies the document specified by the delete command. Next, in box 1209, the process updates the soft delete segment sequence to reflect the newly added soft delete document.

[0103] If the process receives a recovery command, then in box 1210, the process deletes the soft-deleted document from the soft-delete shard that identifies the document specified in the recovery request. Next, in box 1208, the process updates the soft-delete shard sequence to reflect the newly removed soft-deleted document.

[0104] If the process receives a query command, in box 1212, the process executes the query in the primary shard and aggregates the query results. Next, in box 1214, the process determines whether a full index exists that can be used for the soft-deleted shard; if so, the process continues to box 1216; otherwise, the process continues to box 1218. In box 1216, if the process determines that a full index is available in the SDS, the process executes the query in the SDS using the SDS index. On the other hand, in box 1218, if the process determines that a full index is not available in the SDS, the process executes the query in the SDS using a full table scan. In box 1220, the process determines whether the query in box 1216 or box 1218 returns results. If so, the process continues to box 1222; otherwise, the process continues to box 1224. In box 1222, since the same query is executed for both the primary shard and the SDS, and since the documents in the SDS are essentially copies of documents from the soft-deleted primary shard, if the query matches documents from the SDS, the query will also match documents from the soft-deleted primary shard. Therefore, in box 1222, the process removes the query results identified by SDS documents from the aggregated main shard query results that were returned as query results. This action removes soft-deleted documents from the search results because it is not expected to include deleted documents in the query results. Next, in box 1224, the process responds to the query by returning the aggregated main shard query results (lacking any results removed due to SDS matching).

[0105] If the process does not receive a command, then in box 1226, the process checks whether the retention period for any of the SDS documents has expired. If so, the process continues to execute boxes 1228, 1230, 1232, and 1234 for each SDS document whose retention period has expired. In this embodiment, executing boxes 1228, 1230, 1232, and 1234 is equivalent to hard-deleting documents from the sharded database. In box 1228, the process deletes the document identified by the expired SDS document from the primary shard. Next, in box 1230, the process updates the primary shard's index to reflect the deletion of the document from the primary shard. Next, in box 1232, the process updates the SDS index to reflect the deletion of the expired SDS document. Finally, in box 1234, the process deletes the expired SDS document.

[0106] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms “comprising,” “including,” “having,” “containing,” or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such compositions, mixtures, processes, methods, articles, or apparatus.

[0107] Additionally, the term "illustrative" is used herein to mean "serving as an example, illustration, or description." Any embodiment or design described herein as "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" are understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" should be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0108] References to "an embodiment," "embodiment," "example embodiment," etc., in this specification indicate that the described embodiment may include a particular feature, structure, or characteristic; however, each embodiment may or may not include that particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that incorporating other embodiments to affect that feature, structure, or characteristic is within the knowledge of those skilled in the art, regardless of whether it is explicitly described.

[0109] The terms “about,” “substantially,” “approximately,” and variations thereof are intended to include a degree of error associated with a measurement of a specific quantity based on the equipment available at the time of filing this application. For example, “about” may include a range of ±8%, 5%, or 2% of a given value.

[0110] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments described herein.

[0111] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments described herein.

[0112] Therefore, the illustrative embodiments provide a computer-implemented method, system, or apparatus, and a computer program product for managing participation in online communities, as well as other related features, functions, or operations. Where embodiments or portions thereof are described with respect to a type of device, the computer-implemented method, system, or apparatus, computer program product, or portions thereof are adapted or configured for use with appropriate and comparable performance to that type of device.

[0113] Where the embodiments are described as being implemented within an application, the delivery of applications in a Software as a Service (SaaS) model can be envisioned within the scope of the illustrative embodiments. In the SaaS model, the ability to implement an application in an embodiment is provided to a user by executing the application within a cloud infrastructure. Users can access the application using various client devices through thin client interfaces such as web browsers (e.g., web-based email) or other lightweight client applications. Users do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage of the cloud infrastructure. In some cases, users may not even have the ability to manage or control the SaaS application. In some other cases, the SaaS implementation mechanism of the application may allow for possible exceptions to limited user-specific application configuration settings.

[0114] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0115] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0116] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0117] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0118] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0119] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0120] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0122] Embodiments of the present invention can also be delivered as part of a service agreement with client companies, non-profit organizations, government entities, internal organizational structures, etc. Aspects of these embodiments may include configuring computer systems to perform, and deploying, some or all of the software, hardware, and web services implementing the methods described herein. Aspects of these embodiments may also include analyzing client operations, creating recommendations in response to the analysis, building a system to implement portions of the recommendations, integrating the system into existing processes and infrastructure, metering system usage, allocating fees to users of the system, and billing for system usage. Although the above embodiments of the invention have been described by setting forth their respective advantages, the invention is not limited to their specific combinations. Rather, these embodiments can be combined in any manner and number according to the intended deployment of the invention without losing their beneficial effects.

Claims

1. A computer-implemented method, comprising: Receives a request to delete a specified document from the location of the primary shard in the sharded database; A soft delete document is inserted into a soft delete segment, wherein the soft delete document identifies the specified document in a position separate from the main segment, and the specified document is retained in the position of the main segment; Receive a query from the client application, wherein the specified document satisfies the query; and Prevent the specified document from being returned in response to the query, while the soft-deleted document associated with the specified document remains in the soft-deleted fragment.

2. The computer-implemented method according to claim 1, wherein, The sharded database is a NoSQL database.

3. The computer-implemented method according to claim 1 further includes: Receive a recovery request to restore the specified document to the sharded database; as well as In response to the recovery request, the specified document is restored to the shard database, wherein the recovery includes removing the soft-deleted document from the soft-deleted shard.

4. The computer-implemented method according to claim 1, wherein, The soft-deleted document includes a unique identifier (UID) for the specified document and a pointer to the specified document.

5. The computer-implemented method according to claim 4, further comprising: The query is executed for the master data shards.

6. The computer-implemented method according to claim 5, further comprising: Execute the query for the soft-delete shard; as well as The search results for soft deletions are identified by detecting whether the soft-deleted documents returned from a query targeting the soft-deleted shard match the specified documents returned from a query targeting the master data shard.

7. The computer-implemented method according to claim 6, wherein, The search results that identify the soft delete include: The system detects whether the UID of the soft-deleted document returned from the query for the soft-deleted shard matches the UID of the specified document returned from the query for the master data shard.

8. The computer-implemented method according to claim 6, further comprising: In response to the search results that identify the soft deletion, the specified document returned from the query targeting the master data shard is removed.

9. The computer-implemented method according to claim 5, further comprising: In response to inserting the soft-deleted document into the soft-deleted shard, a soft-delete index for the soft-deleted document in the soft-deleted shard is created.

10. The computer-implemented method according to claim 9, further comprising: In response to receiving the query, it is determined that the soft-deleted index is incomplete; as well as In response to the determination that the soft-delete index is incomplete, a full table scan is used to execute the query against the soft-delete shard.

11. The computer-implemented method according to claim 10, further comprising: In response to receiving the query, determine that the soft-deleted index is complete; as well as In response to determining that the soft-delete index is complete, the query is executed against the soft-delete shard using the soft-delete index.

12. The computer-implemented method according to claim 1, further comprising: The time elapsed since receiving the request to delete the specified document has reached the prescribed retention period; as well as Perform a hard delete procedure to remove the specified document from the sharded database.

13. The computer-implemented method according to claim 12, wherein, The hard delete process includes: Delete the specified document from the primary shard of the sharded database; Update the index of the primary shard to reflect the deletion of the specified document from the primary shard; Update the soft delete index of the soft delete shard to reflect the deletion of the soft delete document from the soft delete shard; and Delete the soft-delete document from the soft-delete segment.

14. A computer program product comprising one or more computer-readable storage media and program instructions commonly stored on the one or more computer-readable storage media, the program instructions being executable by a processor to cause the processor to perform operations, the operations including: Receives a request to delete a specified document from the location of the primary shard in the sharded database; A soft delete document is inserted into a soft delete segment, wherein the soft delete document identifies the specified document in a position separate from the main segment, and the specified document is retained in the position of the main segment; Receive a query from the client application, wherein the specified document satisfies the query; and Prevent the specified document from being returned in response to the query, while the soft-deleted document associated with the specified document remains in the soft-deleted fragment.

15. The computer program product according to claim 14, wherein, The stored program instructions are stored in a computer-readable storage device in the data processing system, and the stored program instructions are transmitted from a remote data processing system via a network.

16. The computer program product according to claim 14, wherein, The stored program instructions are stored in a computer-readable storage device in the server data processing system, and wherein the stored program instructions, in response to a request, are downloaded via a network to a remote data processing system for use in a computer-readable storage device associated with said remote data processing system, further comprising: Program instructions for measuring the use of the program instructions associated with the request; and Program instructions used to generate bills based on measured usage.

17. The computer program product according to claim 14, further comprising: Receive a recovery request to restore the specified document to the sharded database; as well as In response to the recovery request, the specified document is restored to the shard database, wherein the recovery includes removing the soft-deleted document from the soft-deleted shard.

18. The computer program product according to claim 14, further comprising: Execute the query on the master data shards; Execute the query for the soft-delete shard; as well as The search results for soft deletions are identified by detecting whether the soft-deleted documents returned from a query targeting the soft-deleted shard match the specified documents returned from a query targeting the master data shard.

19. A computer system comprising a processor and one or more computer-readable storage media, and program instructions commonly stored on the one or more computer-readable storage media, the program instructions being executable by the processor to cause the processor to perform operations, the operations including: Receives a request to delete a specified document from the location of the primary shard in the sharded database; A soft delete document is inserted into a soft delete segment, wherein the soft delete document identifies the specified document in a position separate from the main segment, and the specified document is retained in the position of the main segment; Receive a query from the client application, wherein the specified document satisfies the query; and Prevent the specified document from being returned in response to the query, while the soft-deleted document associated with the specified document remains in the soft-deleted fragment.

20. The computer system of claim 19, further comprising: Receive a recovery request to restore the specified document to the sharded database; as well as In response to the recovery request, the specified document is restored to the shard database, wherein the recovery includes removing the soft-deleted document from the soft-deleted shard.

Images