Granular change detection in distributed storage systems

Abstract

Disclosed herein is a technique for granular change detection in a distributed storage system. In one embodiment, a method includes: receiving a subset of data representing a new version of a document having multiple attributes. The subset of data includes hash values ​​corresponding to the attributes of the document. The method also includes: for each of the attributes, performing a comparison of the hash value in the new version with the hash value of the attribute in a previous version of the document. Based on the comparison performed, determining whether the attribute has changed in the new version of the document. The method may then include: sending a notification to one or more computing services that were previously registered to receive notifications about changes to an identified one of the attributes, without sending the notification to other computing services that were not registered to receive notifications.

Description

Background Art

[0001] Distributed storage systems typically include routers, switches, bridges, and other network devices that interconnect a large number of computer servers, network storage devices, and other types of computing devices via wired or wireless network links. A computer server may host one or more virtual machines, containers, or other types of virtualized components to provide users with various computing and / or storage services. For example, a computer server may be configured to provide data storage and retrieval services that allow users to store, edit, retrieve, or perform other data management tasks. Summary of the invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Today's distributed storage systems are typically arranged according to a layered architecture to provide reliable and scalable storage services to a large number of users. Each layer of the distributed storage system can rely on a corresponding computing system to provide the designed service. For example, a distributed storage system may include a storage layer that is highly scalable and capable of storing large amounts of data. The storage layer may typically include a collection of back-end servers that are configured to facilitate storage, retrieval, and / or other data management tasks for users. The collection of back-end servers may also provide computing resources for performing various data analysis or analytical tasks. Examples of such analytical tasks include aggregation of document views, detection of modification signals, calculation of trending documents, etc. An example of such a storage layer is Microsoft Outlook 2016 provided by Microsoft Corporation of Redmond, Washington.

[0004] Although the storage layer is suitable for performing storage and analysis of large amounts of data, the data structures used in the storage layer may not be suitable for providing random access to individual data items. For example, the storage layer may not access data items in a list at a constant time that is independent of the position of the data item in the list or the size of the list. In addition, the scalability of such distributed storage systems is often achieved by splitting or "sharding" the stored data or the index to such data into partitions. For example, in some distributed storage systems, the main index of a data item in the storage layer can be split into multiple sub-indexes. In contrast to referencing a data item, a main index can reference a sub-index, which in turn can reference a data item. In such a distributed storage system, when a user requests a stored data item, multiple operations (e.g., fan-out federation) are performed in the storage layer to resolve such a reference. Performing multiple operations may result in high latency and computational load in serving user requests.

[0005] A technique for providing fast random access to stored data is to cache a subset of the stored data (sometimes referred to as "high-impact data") in a centralized repository that is configured to provide a cache service for high-impact data. For example, a collection of analysis servers and / or computing processes (referred to herein as "ingestion processors") running in a storage layer can be configured to push a subset of the stored data as high-impact data to a centralized repository. The high-impact data pushed to the centralized repository can include various categories of stored data that may be of interest to various scenario computing services. In one example, the high-impact data for documents stored in the storage layer cached in the centralized repository can include the document name, document extension (e.g., "txt"), the date / time of the last update, the number of times the document has been viewed, the number of times the document has been modified, the universal resource locator (URL) from which the document can be retrieved, and / or other suitable information related to the document. By caching high-impact data, users can easily retrieve documents and / or other relevant information about the document from the centralized repository.

[0006] The scene computing service can be configured to retrieve cached high-impact data from a centralized repository and react to changes in the underlying data stored in the storage layer to provide a corresponding user experience. For example, in the document example above, the search index service may be interested in changes to the title, body, appendix, or other suitable types of document content for updating the search index. On the other hand, the document metadata service may be interested in the document's viewer list and the document's last modified timestamp for updating the number of views / edits of the document.

[0007] In some distributed storage systems, once any change is detected in a stored document, a blanket notification is sent to all scene computing services for further processing. Examples of detected changes may include write operations to the attributes of a document. However, such a blanket notification may result in a waste of computing resources in a distributed storage system. First, the transmission of a blanket notification may involve resource-intensive notification calls (e.g., hypertext transfer protocol or "HTTP" requests or remote procedure calls). Second, different scene computing services may be interested in changes to different attributes of a document. For example, in the document example, the search index service may not be interested in changes to the viewer list of a document used to update the search index or the number of modifications to a document. On the other hand, the document metadata service may not be interested in changes to the body or title of a document. Therefore, the changes indicated in the blanket notification may apply to some scene computing services, but not to other scene computing services. Even if it is not related to the changes indicated in the blanket notification, all scene computing services may consume additional computing, network, input / output, and / or other suitable types of resources to determine whether the changes indicated in the blanket notification are related to individual scene computing services. For example, various types of resources may be consumed to perform additional read operations on high-impact data at a centralized repository. Such read operations may result in high computational loads on servers hosting the centralized repository, high network bandwidth consumption in a computer network, and / or other negative impacts on the performance and / or operation of the distributed storage system.

[0008] Several embodiments of the disclosed technology can address at least some aspects of the aforementioned deficiencies by implementing granular change detection of document attributes in a distributed storage system. In certain implementations, the distributed storage system may include a storage layer having one or more back-end servers, one or more centralized repository servers, and one or more scene computing servers operably coupled to each other via a computer network. The centralized repository server may be configured to provide centralized repository and cache services for a subset of the data stored in the storage layer. The scene computing server may be configured to provide various types of computing services, such as the search index service and document metadata service described above, in response to changes in certain attributes of documents stored in the back-end storage server.

[0009] The back-end server may be configured to provide various data storage services to facilitate storage, retrieval, modification, deletion, and / or other suitable types of data management operations. The back-end server may also be configured to provide an ingestion processor configured to analyze new and / or updated data received at the storage layer and to be stored in the storage layer. In some implementations, the ingestion processor may be configured to receive a registration or other suitable type of indication from a scene computing service. A registration individually indicates changes to one or more attributes of a document that the corresponding scene computing service wishes to receive notification of. Such a registration may be maintained in a data repository as a list, table, or other suitable data structure.

[0010] In some embodiments, the centralized repository may include a change detector configured to perform granular change detection on one or more attributes of data items received at the ingestion processor. Although the change detector is described below as part of the centralized repository, in other embodiments, the change detector may also be configured as part of the ingestion processor, as a standalone computing service in a distributed storage system, or other suitable configuration.

[0011] Using a document as an example, when a new version of a document is received at an ingestion processor, the ingestion processor may extract certain high-impact data from the document and write the extracted high-impact data to a centralized repository. Furthermore, a change detector at the centralized repository (or a component thereof) may be configured to compare various values ​​of attributes of the document in the received new version with various values ​​of attributes in a previous version stored in the centralized repository. For example, the change detector may compare the body and / or title of the document to determine whether a change is detected in the body and / or title. In another example, the change detector may also compare the number of views and / or modifications of the document in the new version with the number of views and / or modifications of the document in the old version.

[0012] In response to detecting a change in at least one of the attributes, the change detector can be configured to provide an indication of the detected change to the ingestion processor. In response, the ingestion processor can be configured to generate a notification and send the notification to those scene computing services that have registered to receive notifications of detected changes to the attributes. For example, when a change in the body and / or title of a document is detected, the ingestion processor can be configured to send a notification to a search index service rather than a document metadata service. On the other hand, when a change in the number of views or the number of modifications is detected, the ingestion processor can be configured to send a notification to a document metadata service rather than a search index service. Therefore, the number of notifications sent by the ingestion processor and subsequent read operations used to verify the relevance of the notifications can be reduced, thereby resulting in lower computational load, lower network bandwidth consumption, and / or lower consumption of other resources in the distributed storage system.

[0013] In some embodiments, detecting which one or more of the attributes have changed may include associating a hash function with the attributes of the document and executing the associated hash function at the ingestion processor to facilitate determining whether the corresponding attributes have changed. For example, the ingestion processor may be configured to apply the hash function associated with each attribute to the value of the attribute in the received new version of the document to derive a hash value (referred to herein as a "new hash value" or H new ). The ingestion processor can then push the generated new hash value to the centralized repository as part of the high-impact data.

[0014] Upon receiving a new hash value, the change detector at the centralized repository may be configured to compare the new hash value with a previous value of the same attribute of a document stored in the centralized repository (referred to herein as a “previous hash value” or H old ). If the new hash value is different from the previous hash value, the change detector can indicate to the ingest processor that a change in an attribute of the document was detected. If the new hash value matches or substantially matches the previous hash value (e.g., a match greater than 90%), the change detector can indicate that the corresponding attribute has not changed.

[0015] Executing the hash function in the ingestion processor can take advantage of the high scalability of the storage layer to reduce the risk of computing resources in the storage layer becoming a bottleneck. In addition, by selecting a hash function suitable for each attribute (as described in more detail below), the consumption of computing resources at the storage layer can be further reduced, and the derived hash value can have a small data size. The small data size of the generated value can result in low storage overhead in the centralized repository, low network bandwidth consumption, and efficient hash comparison when determining whether an attribute has changed.

[0016] Aspects of the disclosed technology are also directed to selecting appropriate hash functions to associate with various attributes of a stored data item (e.g., the document described above). In some implementations, multiple different hash functions can be statically associated with corresponding attributes of a document. For example, an identification function that returns an input value as an output can be used for the number of times a document has been viewed and the number of times it has been modified. In another example, a xxHash function can be associated with the body of a document.

[0017] In another implementation, the association of a hash function with an attribute can be dynamic, for example, based on the data type and / or size of the value of the attribute. For example, an identification function can be used for an attribute whose value is an integer or a short string (e.g., less than ten or other suitable number of characters). Therefore, the number of times a document is viewed and the number of times it is modified can be associated with an identification function. For values ​​that are not integers or short strings, other hash functions are configured to map the data of the value of the attribute to a hash value with a fixed size that is different from the data size of the value. For example, the Fowler-Noll-Vo (FNV) hash function can be associated with an attribute with a value of four to twenty bytes in data size. The xxHash64 function can be associated with an attribute with a value of a large data size (e.g., greater than twenty bytes). The secure hash heuristic 256 (SHA256) function can be applied to the value of an attribute when a conflict of the same data may not be tolerated (e.g., when the number of changes to be detected is higher than a threshold). For example, when nearly every change to an attribute needs to be detected, applying SHA256 to the attribute's value may produce a 256-bit hash value (or other threshold bit hash value) that has a much lower risk of producing a collision.

[0018] In some implementations, the ingest processor may be configured to utilize a configuration file that associates a hash function with a range of value sizes for an attribute. For example, attributes with values ​​less than or equal to four bytes may be associated with an identification function, while attributes with values ​​between five and twenty bytes may be associated with an FNV hash function. Attributes with values ​​greater than twenty bytes may be associated with xxHash64. Additionally, attributes that cannot tolerate collisions under any circumstances may be associated with SHA256.

[0019] In other implementations, the ingestion processor can be configured to present the dynamic selection of a hash function as an optimization operation using a cost function. Different hash functions have different storage footprints (e.g., output hash value sizes), computational resource costs (e.g., computational costs of computing hash values), collision rates, and / or other suitable characteristics. Thus, the optimization operation can be defined as: for an attribute, select a hash function that minimizes processor cycles and storage overhead while keeping the number of collisions below a threshold and minimizing the total amount of attribute data that must be read for a full comparison. Example cost function J( Hi ) can be expressed as follows:

[0020] J( Hi )=W Collision *CR Hi +W CPU *CPU Hi +W Storage *Storage Hi +W Data *Data Hi , where CR Hi is the conflict rate, CPU Hi is the processor cycles, Storage Hi is the storage size of the hash value, Data Hi ∈{PropertySize,Storage Hi} and W is a weight associated with various corresponding parameters. When the distributed storage system is in operation, the different weights associated with the aforementioned parameters may change. For example, the available system resources (e.g., computing or storage) may change so that computing resources become abundant. Therefore, the weights for computing resources may be adjusted so that a different hash function is selected for the same attribute than before the available system resources change.

[0021] Additional aspects of the disclosed technology are directed to performing change detection by grouping related attributes together and generating a hash value for the group of attributes rather than for each attribute in the group. Without being bound by theory, it is recognized that certain attributes of a data item (e.g., a document) frequently change together. For example, the number of times a document is viewed often changes with the number of times the document is modified. Therefore, grouping attributes into groups and generating a hash value for the group can allow for a quick determination of whether any of the attributes in the group have changed. If no change to the group is detected, a full comparison of each of the attributes in the group can be skipped, and metadata can be added to a new version indicated accordingly. If a change to the group is detected, each of the attributes in the group can be compared to see which one or more of the attributes have changed.

[0022] According to additional aspects of the disclosed technology, grouping attributes can also utilize optimization operations. If the correlation of changes between different attributes is used to group the attributes, a "grouping correlation threshold" can be set so that the amount of data to be read for attribute and hash comparison is below the size threshold and / or minimized. The correlation of the change can include data indicating the probability (e.g., as a percentage) that the first attribute may change when the second attribute changes, and vice versa. The correlation of the change can be developed via statistical analysis of historical data of the attribute or via other suitable means. In some embodiments, if the value of the attribute has been hashed, a combination operation (e.g., left shift and perform bitwise XOR) can be used to derive a group hash value. However, if the attribute has been passed through an identification function, an additional hash operation can be performed on the combination of the value of the attribute (referred to as the group value) to derive the group hash value. In other embodiments, multiple groupings of attributes can be used, wherein the attribute can be part of multiple groups.

[0023] A technique for generating attribute groupings may include maintaining a covariance matrix for the attributes. A covariance matrix is ​​a matrix whose elements in the i, j position are the covariances (i.e., measures of joint variability) between the i-th and j-th elements of a random vector. The computation of the covariance matrix can be simplified such that if an attribute has changed, the value of the attribute is 1. If the attribute has not changed, the value of the attribute is 0. If an attribute is not identical to an attribute group P already in a cluster, the covariance matrix is ​​0. Cj Some (or all) of the attributes in <Pi,P Cj >corresponding value is above threshold Tc), a clustering heuristic can be used to construct groupings by assigning attributes Pi to existing clusters (attribute groups). If the attributes are not relevant enough (i.e., the value for attribute Pi in the covariance matrix is ​​not above threshold Tc), a new cluster (i.e., a new attribute group) containing only the attribute can be created. Another grouping heuristic can include determining the "closest" cluster with respect to a cluster similarity function, and assigning the attributes to the closest cluster (i.e., attribute group). Regardless of which heuristic is used to create attribute groupings, the amount of data to be read and compared in order to detect which attributes have changed can be maintained below a threshold and / or minimized.

[0024] In some implementations, the attribute groupings may be calculated before the distributed storage system operates, but the attribute groupings may also be calculated periodically in an offline batch mode. After the attribute groupings are determined, a bit field in which each bit corresponds to an attribute may be used to record which attribute has changed. For example, if the corresponding attribute has not changed, the bit may be set to zero. When the corresponding attribute has changed, the same bit may be set to one. In other implementations, the attribute groupings may be changed in an online manner to better accommodate temporal changes in data. In other implementations, a single attribute may be included in multiple different groups. This inclusion may allow for rapid change detection. For example, when a change is detected in one group but not in another group, it may be determined that the attributes in both groups have not changed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram illustrating a distributed storage system implementing granular change detection according to an embodiment of the disclosed technology.

[0026] Figure 2A-2D is a diagram showing an embodiment of the disclosed technology. Figure 1 A schematic diagram of certain hardware / software components of a distributed storage system that performs granular change detection.

[0027] Figure 3A-3C is a diagram showing the steps during signature generation according to an embodiment of the disclosed technology. Figure 2A-2D Schematic diagram of some of the hardware / software components of the signature assembly.

[0028] Figure 4 is a diagram showing an embodiment of the disclosed technology. Figure 2A-2D Schematic diagram of grouping certain hardware / software component implementation properties of a signature component.

[0029] Figure 5 is a Venn diagram illustrating one example grouping of properties according to an embodiment of the disclosed technology.

[0030] Figure 6A-6C is a flow chart illustrating a process for granular change detection in a distributed storage system according to an embodiment of the disclosed technology.

[0031] Figure 7 Is suitable for Figure 1 A computing device that is some component of a distributed storage system. DETAILED DESCRIPTION

[0032] Certain embodiments of systems, devices, components, modules, routines, data structures, and processes for granular change detection in a data center or other suitable distributed storage system are described below. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the relevant art will also understand that the technology may have additional embodiments. It may also be described without the following references. Figure 1-Figure 7 The technology is implemented without any details in the details of the described embodiments.

[0033] As used herein, the term "distributed storage system" generally refers to an interconnected computer system having routers, switches, bridges, and other network devices that interconnect a large number of computer servers, network storage devices, and other types of computing devices to each other and / or to an external network (e.g., the Internet) via wired or wireless network links. Individual servers or storage nodes may include one or more persistent storage devices, such as hard disk drives, solid-state drives, or other suitable computer-readable storage media.

[0034] As also used in this article, "document" generally refers to a computer file that contains various types of data and metadata that describe and / or identify a computer file. Various types of data and metadata are referred to as "properties" of the document in this article. Example properties may include title, content body, author, universal resource locator (URL), number of times viewed, number of times edited, and / or other suitable information related to the document. Each property can be defined as a parameter containing a corresponding value. For example, the author property can have a value containing a name (e.g., "Nick Jones"). The number of times viewed property can include a value containing an integer (e.g., "4").

[0035] As used herein, a "version" of a document generally refers to a state or form of a document. A version may be defined by a version number, edit date / time, editor name, saved date / time, and / or other suitable attributes. A "new version" generally refers to a state of a document in which there is at least one change to any aspect of the document's data and / or metadata. For example, a new version of a document may include an edit to the content of the document or a change to one or more properties of the document (e.g., the number of times it has been viewed). A "previous version" is another state of a document from which the new version may be derived, directly or indirectly.

[0036] As also used in this article, "scenario computing services" generally refer to one or more computing resources provided by a computer network (e.g., the Internet) designed to provide a designed user experience. Example scenario computing services include software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). SaaS is a software distribution technology in which software applications are hosted by a cloud service provider, such as in a data center, and accessed by users through a computer network. PaaS generally refers to the delivery of operating systems and associated services through a computer network without requiring download or installation. IaaS generally refers to outsourcing devices for supporting storage devices, hardware, servers, network devices, or other components, all of which can be accessed through a computer network.

[0037] Various scene computing services can be configured to provide different user experiences. An example scene computing service can be configured to compile various attributes of a document into a search index. In another example, another scene computing service can be configured to receive a signal indicating that a user viewed a document, and in response to the received signal, provide an electronic message to other users to recommend viewing the same document. In further examples, the scene computing service can be configured to provide other suitable user experiences.

[0038] In some distributed storage systems, once any change is detected in a document, a comprehensive notification is sent to all scene computing services for further processing. However, such comprehensive notifications can lead to a waste of computing resources in the distributed storage system. First, the transmission of the comprehensive notification may involve resource-intensive notification calls (e.g., HTTP requests or remote procedure calls). Second, different scene computing services may be interested in changes to different attributes of a document. For example, a search index service may not be interested in changes to the viewer list of a document used to update the search index or the number of modifications to a document. On the other hand, a document metadata service may not be interested in changes to the body or title of a document. Therefore, the changes indicated in the comprehensive notification may apply to some scene computing services, but have nothing to do with other scene computing services.

[0039] Even if unrelated to the change indicated in the comprehensive notification, all scene computing services may consume additional computing, network, input / output, and / or other suitable types of resources to determine whether the change indicated in the comprehensive notification is related to the individual scene computing services. For example, various types of resources may be consumed to perform additional read operations. Such read operations may result in high computing loads, high network bandwidth in the computer network, and / or other negative effects on the performance and / or operation of the distributed storage system.

[0040] Several embodiments of the disclosed technology may address at least some aspects of the aforementioned deficiencies by implementing granular change detection of document properties in a distributed storage system. In certain implementations, a change detector may be implemented in a distributed storage system to determine which one or more properties of a new version of a document have changed compared to a previous version of the same document. Based on the determined one or more properties, notifications may be selectively sent to scene computing services that were previously registered to receive notifications of changes to one or more properties. Thus, expensive HTTP requests and / or remote procedure calls to scene computing services that are not interested in the detected changes to the properties may be avoided. Moreover, additional computing resources for determining whether the changes indicated in the notification are actually related to the scene computing service may also be avoided. Thus, the computing load, network bandwidth, and / or consumption of other computing resources of the distributed storage system may be reduced compared to utilizing comprehensive notifications, as described below with reference to Figure 1-Figure 7 Described in more detail.

[0041] Figure 1 1 is a schematic diagram showing a computing environment 100 for implementing granular change detection according to an embodiment of the disclosed technology. Figure 1 As shown in FIG. 1 , the computing environment 100 may include one or more client devices 102 of a corresponding user 101, and a distributed storage system 103 interconnected to the client devices 102 via a computer network (not shown) such as the Internet. Figure 1 Specific components of computing environment 100 are shown in the figure, but in other embodiments, computing environment 100 may also include additional and / or different components or arrangements. For example, in some embodiments, computing environment 100 may also include additional network storage devices, network devices, hosts and / or other suitable components (not shown) in other suitable configurations.

[0042] Client devices 102 may each include a computing device that facilitates user 101 to access a cloud storage service provided by distributed storage system 103 via a computer network. In one embodiment, client devices 102 may each include a desktop computer. In other embodiments, client devices 102 may also include a laptop computer, a tablet computer, a smart phone, or other suitable computing device. Although for illustration purposes only, Figure 1 Two users 101 and 101 ′ are shown in FIG. 1 , but in other embodiments, the computing environment 100 may facilitate other suitable numbers of users 101 to access cloud storage or other types of computing services provided by the distributed storage system 103 in the computing environment 100 .

[0043] like Figure 1As shown in FIG. 1 , the distributed storage system 103 may include one or more front-end servers 104, one or more back-end servers 105, a centralized repository 106, and one or more scene servers 108 operably coupled via a computer network 117. The back-end server 105 may be operably coupled to a network storage device 112 configured to store user data. In the example shown, the user data stored at the network storage device 112 is shown as a document 110. In other examples, the user data may also include image files, video files, or other suitable types of computer files. Although in FIG. Figure 1 Specific components of the distributed storage system 103 are shown in FIG. 1 , but in other implementations, the distributed storage system 103 may also include additional and / or different components.

[0044] The computer network 117 may include any suitable type of network. For example, in one embodiment, the computer network 117 may include an Ethernet or Fast Ethernet network with routers, switches, load balancers, firewalls, and / or other suitable network components that implement the RDMA over Converged Ethernet (RoCE) protocol. In other embodiments, the computer network 117 may also include an InfiniBand network with corresponding network components. In other embodiments, the computer network 117 may also include a combination of the above and / or other suitable types of computer networks.

[0045] The front-end server 104 may be configured to interact with the user 101 via the client device 102. For example, the front-end server 104 may be configured to provide a user interface (not shown) on the client device 102 to allow the corresponding user 101 to upload, modify, retrieve or perform other suitable operations on the document 110 stored in the network storage device 112. In another example, the front-end server 104 may also be configured to provide a user portal (not shown) that allows the user 101 to set retention, privacy, access or other suitable types of policies to be implemented on the document 110 in the network storage device 112. In some embodiments, the front-end server 104 may include a web server, a security server and / or other suitable types of servers. In other embodiments, the front-end server 104 may be omitted from the distributed storage system 103 and provided by an external computing system (e.g., a cloud computing system (not shown)).

[0046] The backend server 105 may be configured to provide various data storage services to facilitate storage, retrieval, modification, deletion, and / or other suitable types of data management operations on the documents 110 in the network storage device 112. For example, the backend server 105 may be configured to store the documents 110 and / or indexes to the documents 110 in the shards. The backend server 105 may also be configured to provide facilities for performing fan-out joins on data from the various shards in response to suitable queries from, for example, the user 101. In further examples, the backend server 105 may also be configured to provide facilities for performing duplication reduction, data compression, access control, and / or other suitable operations.

[0047] The backend server may also be configured to provide an ingest processor 107 ( Figure 2A-2D ), the ingestion processor 107 is configured to analyze new and / or updated data of the document 110 received at the network storage device 112 and to be stored in the network storage device 112. In some implementations, the ingestion processor 107 can be configured to receive a registration or other suitable type of indication from the scene server 108 and / or the corresponding scene computing device. The registration can individually indicate changes to one or more attributes of the document 110, the corresponding scene server or the scene computing device, that wish to receive notifications. Such a registration can be provided as a notification list 128 ( Figure 2D ), tables, or other suitable data structures are maintained in other suitable data repositories.

[0048] Although the backend server 105 may be suitable for performing storage and analysis of large amounts of data, the data structures used in the backend server 105 and / or the network storage device 112 may not be suitable for providing random access to individual data items (e.g., documents 110). For example, the backend server 105 may not access a document 110 in a list at a constant time regardless of the position of the document 110 in the list or the size of the list. In addition, when a user requests a stored document, the backend server 105 may perform multiple operations (e.g., fan-out joins) to resolve indirect references. Performing multiple operations may result in high latency and computational load to service user requests.

[0049] Centralized repository 106 may be configured to: Figure 1The ingestion processor 107 running in the backend server 105 may be configured to push a subset of the stored data of the document 110 as data 111 to the centralized repository 106 for storage in the data repository 114.

[0050] The data 111 pushed to the centralized repository 106 may include various categories of stored data that may be of interest to various scene servers 108 and / or scene computing services. In one example, the data cached at the centralized repository 106 for the document 110 stored in the network storage device 112 may include the document name, the document extension (e.g., "txt"), the date / time of the last update, the number of times the document has been viewed, the number of times the document has been modified, the universal resource locator (URL) where the document can be retrieved, and / or other suitable information related to the document. By caching the data 111, the user 101 can easily retrieve the document information and / or other related information from the centralized repository 106.

[0051] The scene server 108 may be configured to retrieve cached data 111 from the centralized repository 106 and react to changes in the underlying data of the document 110 stored in the network storage device 112 to provide a corresponding user experience. For example, the search index service may be interested in changes to the content of the title, body, appendix, or other suitable type of document 110 for updating a search index (not shown). On the other hand, the document metadata service may be interested in the number of views of the document 110 or the list of viewers and the timestamp of the last modification of the document 110 for updating the number of views / edits of the document 110. In response to the change in the number of views, the document metadata service may be configured to send a message 118 to, for example, the user 101' to notify the user 101' that another user 101 has viewed and / or edited the document 110. Although the scene server 108 is in Figure 1 108 is shown as a physical computing device, but in some implementations, the functionality of the scene server 108 can be provided as a corresponding scene computing service by a suitable server (e.g., the front-end server 104) executing suitable instructions.

[0052] In some distributed storage systems, once any change is detected in a stored document 110, a change notification 116 is sent to all sceners 108 for further processing. Examples of detected changes may include write operations to properties of the document 110. However, such comprehensive notifications may result in a waste of computing resources in the distributed storage system 103. First, the transmission of the change notification 116 may involve resource-intensive notification calls (e.g., Hypertext Transfer Protocol or "HTTP" requests or remote procedure calls).

[0053] Second, different scene servers or scene computing services may be interested in changes to different properties of the document 110. For example, the search index service may not be interested in changes to the list of document 110 viewers used to update the search index or the number of modifications to the document 110. On the other hand, the document metadata service may not be interested in changes to the body or title of the document 110. Therefore, the changes indicated in the change notification 116 may be applicable to some scene computing services, but not to other scene computing services. Even if it is not related to the changes indicated in the change notification 116, all scene computing services can consume additional computing, network, input / output and / or other suitable types of resources to determine whether the changes indicated in the change notification 116 are related to the respective scene computing services. For example, various types of resources can be consumed to perform additional read operations on the data 111 at the centralized repository 106. Such read operations can result in high computing loads on the servers hosting the centralized repository 106, high network bandwidth in the computer network 117, and / or other negative effects on the performance and / or operation of the distributed storage system 103.

[0054] Several embodiments of the disclosed technology can address at least some aspects of the aforementioned deficiencies by implementing granular change detection of document properties in a distributed storage system 103. In certain implementations, a change detector 122 can be implemented in a centralized repository 106 to determine which one or more of the properties of a new version of a document 110 have changed compared to a previous version of the same document 110. Based on the determined one or more properties, a change notification 116 can be selectively sent to a scene server 108 that was previously registered to receive notifications of changes to one or more properties. Thus, expensive HTTP requests and / or remote procedure calls to scene computing services that are not interested in the detected changes to the properties can be avoided. Moreover, additional computing resources for determining whether the changes indicated in the change notification 116 are actually related to the scene server 108 can also be avoided. Therefore, compared to utilizing comprehensive notifications, the computing load, network bandwidth, and / or consumption of other computing resources of the distributed storage system 103 can be reduced, as described below with reference to Figure 2A-2D Described in more detail.

[0055] Figure 2A-2D is a diagram showing an embodiment of the disclosed technology. Figure 1 A schematic diagram of certain hardware / software components of the distributed storage system 103 that performs granular change detection. Figure 2A-2D In other figures of this paper, the Figure 1 Certain components of the computing environment 100 and the distributed storage system 103. For example, Figure 2A-2D In the example, for ease of explanation, the front-end server 104 and the computer network 117 are omitted from the distributed storage system 103. Figure 2A-2D In other figures of this article, each software component, object, class, module and routine can be a computer program, process or process written as source code in C, C++, C#, Java and / or other suitable programming languages. Components can include, but are not limited to, one or more modules, objects, classes, routines, attributes, processes, threads, executable files, libraries or other components. Components can be in source or binary form. Components can include aspects of source code (e.g., classes, attributes, processes, routines) before compilation, compiled binary units (e.g., libraries, executable files) or artifacts (e.g., objects, processes, threads) that are instantiated and used at runtime.

[0056] Components within a system may take different forms within the system. As an example, a system including a first component, a second component, and a third component may include, but is not limited to, a system having a first component as a property in source code, a second component as a binary compiled library, and a third component as a thread created at runtime. A computer program, procedure, or process may be compiled into object code, intermediate code, or machine code and presented for execution by one or more processors of a personal computer, a network server, a laptop, a smart phone, and / or other suitable computing device.

[0057] Likewise, a component may include a hardware circuit. One of ordinary skill in the art will recognize that hardware may be viewed as petrified software, and software may be viewed as liquefied hardware. As just one example, the software instructions in a component may be burned into a programmable logic array circuit, or may be designed as a hardware circuit with an appropriate integrated circuit. Likewise, hardware may be emulated by software. Various implementations of source code, intermediate code, and / or object code and associated data may be stored in a computer memory, which may include a read-only memory, a random access memory, a disk storage medium, an optical storage medium, a flash memory device, and / or other suitable computer-readable storage media in addition to a propagated signal.

[0058] like Figure 2AAs shown in FIG. 1 , the back-end server 105 may include a server for receiving data transmitted via the front-end server 104 ( Figure 1 ) from the client device 102 of the user 101. The document 110 may include a plurality of attributes 120. In the illustrated embodiment, the ingestion processor 107 may include an interface component 130, a signature component 132, and a notification component 134 operably coupled to each other. In other embodiments, the ingestion processor 107 may also include network, security, database, or other suitable types of components.

[0059] The interface component 130 may be configured to receive a new version of the document 110 and provide the received document 110 to the signature component 132 for further processing. The interface component 130 may also be configured to store, retrieve, or perform other suitable file operations on the document 110 in the network storage device 112. For example, in the illustrated embodiment, the interface component 130 may be configured to store a copy of the received new version of the document 110 in the network storage device 112 as a separate file from the previous version, or to replace the previous version of the document 110 with the copy.

[0060] The signature component 132 may be configured to generate a signature 121 corresponding to one or more of the attributes 120. In some embodiments, the signature 121 of the attribute 120 may include a hash value of the attribute 120. For example, the signature 121 of the content body of the document 110 may include a random alphanumeric string of a preset length. In other embodiments, the signature of the attribute 120 may include a value of the attribute 120. For example, the value of the number of views attribute of the document 110 may include the integer "4". In other examples, the signature 121 may include other suitable values ​​derived from and / or corresponding to various attributes 120 of the document 110.

[0061] like Figure 2A As shown in , after generating the signature 121, the signature component 132 can instruct the interface component 130 to send the data 111 extracted from the new version of the document 110 together with the signature 121 to the centralized repository 106 for granular change detection. Although the embodiment of granular change detection is described herein as being implemented in the centralized repository 106, in other embodiments, such granular change detection can also be implemented on one or more of the backend server 105 or other suitable servers (not shown) of the distributed storage system 103.

[0062] like Figure 2AAs shown in , the centralized repository 106 may include a change detector 122 and a change indicator 124 operably coupled to each other. The change detector 122 may be configured to determine which one or more of the detected attributes have changed. In some embodiments, the change detector 122 may be configured to compare the respective signatures 121 of a new version of the document 110 with those signatures 121 of a previous version stored in the data repository 114. In response to determining that the signatures 121 of the new version are the same or substantially the same (e.g., the similarity exceeds 90%), the change detector 122 may be configured to indicate that the corresponding attribute 120 of the document 110 has not changed. Otherwise, the change detector 122 may be configured to indicate that the corresponding attribute 120 has changed. Figure 2B As shown in , after iterating through all signatures 121 ( Figure 2A ) thereafter, the change indicator 124 may be configured to send a list of attribute IDs 123 identifying a list of attributes 120 that have changed to the ingest processor 107 at the backend server 105.

[0063] In response to receiving the attribute ID 123, the notification component 134 of the ingest processor 107 can be configured to retrieve the notification list 128 from the network storage device 112. The notification list 128 may include data indicating which one or more of the scene servers 108 (and / or corresponding scene computing services) are interested in changes to a particular attribute 120 of the document 110. The notification list 128 may be generated by receiving registration data (not shown) from the scene servers 108, by assigning default values, or via other suitable techniques. Based on the retrieved notification list 128, the notification component 134 may be configured to determine one or more scene servers 108 to receive the change notification 125 based on the attribute ID 123. In the example shown, the first scene server 108a may be indicated in the notification list 128 as being interested in changes to the attribute 120 identified by one of the attribute IDs 123. Therefore, the change notification 116 is only sent to the first scene server 108a, and not to the second scene server 108b and the third scene server 108c.

[0064] like Figure 2C As shown in FIG. 1 , after receiving the change notification 116, the first scene server 108a may be configured to provide a message 118 to the user 101' to, for example, notify the user 101' of the detected change of the attribute 120 of the document 110, or to provide other suitable user experiences. Figure 2D, when the user 101 provides another version of the document 110' with different changes to one or more of the properties 120', the notification component 134 can be configured to provide a change notification 116' to the third scene server 108c instead of the first scene server 108a or the second scene server 108b when a change to the different property 120' is detected (as indicated by the property ID 123'). Therefore, resource-intensive notification calls to one or more of the scene servers 108 and additional read operations by such one or more scene servers 108 can be avoided.

[0065] Figure 3A-3C is a diagram showing the steps during signature generation according to an embodiment of the disclosed technology. Figure 2A-2D A schematic diagram of some of the hardware / software components of the signature component 132. Figure 3A As shown in , the signature component 132 may include a function selector 142 and a signature generator 146 operatively coupled to each other. The function selector 142 may be configured to select a function for generating the signature 121 based on the value of the attribute 120. As described in more detail below, the function selector 142 may be configured to select a function in a variety of ways. The signature generator 146 may be configured to apply a selected function (e.g., a hash function) to the value of the attribute 120 to generate the signature 121.

[0066] Aspects of the disclosed technology are directed to selecting appropriate functions associated with various attributes 120 of a document 110 in order to transmit the attributes to a backend server 104 ( Figure 1 ) is maintained below a threshold or even minimized. In some implementations, such as Figure 3A , multiple different hash functions may be statically assigned and associated with corresponding attributes 120 of document 110, as indicated by function list 137 stored in network storage device 112. Identification functions that return an input value as an output may be used for times of viewing and times of modification of a document, since the cost of computing hash values ​​for these values ​​may outweigh the savings in computing resources used to compare these values. In another example, the xxHash function may be associated with the body of document 110, since the body may include a large amount of data.

[0067] In other implementations, the association of the hash function with the attribute 120 may be dynamic, for example, based on the data type and / or size of the value of the attribute 120 or other suitable function criteria 138, such as Figure 3B. For example, the function selector 142 may be configured to determine the data type and / or size of the value of the attribute 120, and assign an identification function for an attribute 120 whose value is an integer or a short string (e.g., less than ten or other suitable number of characters) according to the function standard 138. Thus, the number of times a document has been viewed and the number of times it has been modified may be associated with an identification function. In other examples, the function selector 142 may be configured to assign the Fowler-Noll-Vo (FNV) hash function to an attribute 120 having a value of four to twenty bytes in size according to the function standard 138. In another example, the function selector 142 may also be configured to assign the xxHash64 function to an attribute 120 having a value of a large data size (e.g., greater than twenty bytes) according to the function standard 138. In another example, the function selector 142 may be configured to assign the secure hash heuristic 256 (SHA256) function to an attribute 120 having a value that may produce a hash value collision and when such a collision cannot be tolerated.

[0068] In another implementation, Figure 3C As shown in , function selector 142 can be configured to use cost function 139 to present a dynamic selection of a hash function as an optimization operation. Different hash functions can have different storage footprints (e.g., output hash value sizes), computational resource costs (e.g., computational costs of computing hash values), collision rates, and / or other suitable characteristics. Thus, an optimization operation can be defined as: for attribute 120, select a hash function that minimizes processor cycles and storage overhead while keeping the number of collisions below a threshold and minimizing the total amount of attribute data that must be read for a full comparison. Example cost function J( Hi ) can be expressed as follows:

[0069] J( Hi )=W Collision *CR Hi +W CPU *CPU Hi +W Storage *Storage Hi +W Data *Data Hi , where CR Hi is the conflict rate, CPU Hi is the processor cycles, Storage Hi is the storage size of the hash value, Data Hi ∈{PropertySize,Storage Hi} and W is the weight associated with the various corresponding parameters. When the distributed storage system 103 is in operation, the different weights associated with the aforementioned parameters may change. For example, the available system resources may change so that computing resources become abundant. Therefore, the weights for computing resources may be adjusted so that a different hash function is selected for the same attribute 120 than before the available system resources change.

[0070] Figure 4 is a diagram showing an embodiment of the disclosed technology. Figure 2A-2D A schematic diagram of a grouping of certain hardware / software component implementation properties of a signature component. Figure 4 As shown in FIG. 1 , the signature component 132 may also include a grouping module 140 configured to: group some of the attributes 120 of the document 110 into a set; generate a signature 121 of the set; and allow the centralized repository 106 ( Figure 2A ) at the change detector 122 ( Figure 2A ) efficiently determine which one or more of attributes 120 have changed.

[0071] In some implementations, the grouping module 140 can be configured to group related attributes 120 together and generate the signature 121 for a set of attributes 120 rather than for each attribute 120 in the set. Without being bound by theory, it is recognized that certain attributes 120 of the document 110 frequently change together. For example, the number of times a document is viewed often changes with the number of times the document is modified. Therefore, grouping the attributes 120 into a group or set and generating a hash value for the set can allow for a quick determination of whether any of the attributes 120 in the set have changed. For example, if no change to the set is detected, the change detector 122 can skip a full comparison of each of the attributes 120 in the set. If a change to the set is detected, each of the attributes 120 in the set can be compared to see which one or more of the attributes 120 have changed based on the function list 137, the function criteria 138, or the cost function 139, as described above for Figure 3A-3C as described.

[0072] According to additional aspects of the disclosed technology, grouping of attributes 120 can also utilize optimization operations. If the changing correlation between different attributes 120 is used to group the attributes 120, a "grouping correlation threshold" can be set so that the amount of data to be read for attribute 120 and hash comparison is maintained below a size threshold and / or minimized. In some embodiments, if the values ​​of the attributes 120 have been hashed, a combination operation (e.g., left shift and performing a bitwise XOR) can be used to generate a group hash value as the signature 121 of the group. However, if the attributes 120 have been passed through an identification function, additional hash operations can be performed on the combination of the values ​​of the attributes to generate the group hash value. In other embodiments, multiple groupings of attributes 120 can be used, wherein the attributes 120 can be part of multiple groups or sets.

[0073] One technique for generating attribute groupings may include maintaining a covariance matrix (not shown) for the attributes 120. The calculation of the covariance matrix may be simplified such that if the attribute 120 has changed, the value of the attribute 120 is 1. If the attribute 120 has not changed, the value of the attribute 120 is 0. If the attribute 120 is not in the same cluster as the attribute group P already in the cluster, the covariance matrix may be calculated. Cj Some (or all) of the attributes 120 in <Pi,P Cj >corresponding value is above threshold Tc), a clustering heuristic may be used to construct groupings by assigning attributes 120Pi to existing clusters (attribute groups). If attribute 120 is not relevant enough (i.e., the value for attribute Pi in the covariance matrix is ​​not above threshold Tc), a new cluster (i.e., a new attribute group) containing only that attribute may be created. Another grouping heuristic may include determining the "closest" cluster with respect to a cluster similarity function, and assigning attribute 120 to the closest cluster (i.e., attribute group). Regardless of which heuristic is used to create attribute groupings, the amount of data to be read and compared in order to detect which attributes have changed may be maintained below a threshold and / or minimized.

[0074] In some implementations, the attribute groupings may be calculated prior to the distributed storage system 103 operating, but may also be calculated periodically in an offline batch manner or in other suitable processing manners. After the attribute groupings are determined, a bit field in which each bit corresponds to an attribute 120 may be used to record which attribute 120 in the group has changed. For example, if the corresponding attribute 120 has not changed, the bit may be set to zero. When the corresponding attribute has changed, the same bit may be set to one. In other implementations, the attribute groupings may be changed in an online manner to better accommodate temporal changes in data.

[0075] In other implementations, a single attribute 120 may be included in multiple different attribute groups. This inclusion may allow for rapid change detection. For example, Figure 5 As shown in the figure, the first group 150a may include four attributes 120 represented as circles, and the second group 150b may also include four attributes 120'. The first group 150a and the second group 150b share common attributes 120". Therefore, when a change in the first group 150a is detected but a change in the second group 150b is not detected, it can be determined that the shared attributes 120" in both the first group 150a and the second group 150b have not changed.

[0076] Figure 6A-6C is a flow chart illustrating a process for granular change detection in a distributed storage system according to an embodiment of the disclosed technology. Figure 1 Embodiments of the processes are described in the context of computing environment 100, but in other embodiments, the processes may also be implemented in other computing environments having additional and / or different components.

[0077] like Fig. 6A As shown in FIG. 2 , process 200 may include receiving a new version of a document at stage 202. In some embodiments, the new version of a document may have a new version stored in distributed storage system 103 ( Figure 1 ) in the corresponding previous version. In other embodiments, the previous version of the document may not exist in the distributed storage system 103. Then, the process 200 may include comparing various attributes of the received new version of the document at stage 204. Various techniques for comparing attributes may be applied. Figure 2A-2D And reference below Figure 6B Certain example techniques are described. Process 200 may also include identifying, at stage 206, properties whose values ​​have changed in the received new version of the document. After identifying the properties whose values ​​have changed, process 200 may proceed to send a notification to a scene server or scene computing service that was previously registered to receive notifications about changes to the properties.

[0078] Figure 6B An example operation of comparing various attributes of a document is shown. Figure 6B As shown in , the operation may include comparing signatures (e.g., hash values) of attribute values ​​210, which includes identifying the values ​​of the respective attributes at stage 212. The operation may optionally include selecting a function for generating a signature for each attribute at stage 214. Figure 3A-3CVarious example techniques for selecting a function are described. Operations may then include applying the selected function to the value of the corresponding attribute to generate a signature at stage 216. Based on the generated signature, operations may include detecting changes in one or more attributes at stage 218.

[0079] Figure 6C An example operation of detecting attribute changes by grouping attributes into sets is shown. Figure 6C As shown in , the operation may include grouping the attributes into sets at stage 222. Figure 4 Example techniques for grouping attributes are described. Operations may then include generating a group signature for each of the sets at stage 224. For each of the sets, operations may include a decision stage 224 for determining whether a change has been detected in the set based on the group signature. In response to determining that a change has not been detected in the set, operations may include marking all attributes in the set as unchanged. In response to determining that a change has been detected in the set, operations may include determining, at stage 228, whether a change has been detected in the set by performing a decision such as described above with reference to Figure 6B Operations such as those described above are used to detect changes to one or more attributes in a collection.

[0080] Figure 7 Is suitable for Figure 1 The computing device 300 may be adapted to include certain components of the computing environment 100. For example, the computing device 300 may be adapted to include certain components of the computing environment 100. Figure 1 The computing device 300 includes a client device 102, a front-end server 104, a back-end server 105, a centralized repository 106, and a scene server 108. In a very basic configuration 302, the computing device 300 may include one or more processors 304 and a system memory 306. A memory bus 308 may be used to communicate between the processor 304 and the system memory 306.

[0081] Depending on the desired configuration, the processor 304 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 304 can include one or more levels of cache (e.g., a level 1 cache 310 and a level 2 cache 312), a processor core 314, and registers 316. An example processor core 314 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 318 can also be used with the processor 304, or in some implementations, the memory controller 318 can be an internal part of the processor 304.

[0082] Depending on the desired configuration, system memory 306 may be of any type, including but not limited to volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, flash memory, etc.), or any combination thereof. System memory 306 may include an operating system 320, one or more applications 322, and program data 324. The basic configuration 302 described in Figure 7 These components are shown within the inner dashed lines.

[0083] The computing device 300 may have additional features or functions, as well as additional interfaces for facilitating communication between the basic configuration 302 and any other devices and interfaces. For example, a bus / interface controller 330 may be used to facilitate communication between the basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. The data storage device 332 may be a removable storage device 336, a non-removable storage device 338, or a combination thereof. Examples of removable storage devices and non-removable storage devices include magnetic disk devices (e.g., floppy disk drives and hard disk drives (HDDs)), optical disk drives (e.g., compact disk (CD) drives or digital versatile disk (DVD) drives), solid state drives (SSDs), and tape drives, to name a few. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The term "computer-readable storage medium" or "computer-readable storage device" does not include propagated signals and communication media.

[0084] System memory 306, removable storage device 336, and non-removable storage device 338 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 300. Any such computer-readable storage media may be part of computing device 300. The term "computer-readable storage media" does not include propagated signals and communication media.

[0085] The computing device 300 may also include an interface bus 340 for facilitating communication from various interface devices (e.g., output devices 342, peripheral interfaces 344, and communication devices 346) to the basic configuration 302 via the bus / interface controller 330. Example output devices 342 include a graphics processing unit 348 and an audio processing unit 350, which may be configured to communicate with various external devices such as a display or speakers via one or more A / V ports 352. Example peripheral interfaces 344 include a serial interface controller 354 or a parallel interface controller 356, which may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 358. Example communication devices 346 include a network controller 360, which may be arranged to facilitate communication with one or more other computing devices 362 over a network communication link via one or more communication ports 364.

[0086] A network communication link can be an example of a communication medium. Communication media can typically be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier waveform) or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal that has one or more characteristics in its characteristic set, or is changed in a manner that encodes the information in the signal. By way of example and not limitation, communication media can include wired media such as a wired network or a direct line connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0087] The computing device 300 may be implemented as part of a small form factor portable (or mobile) electronic device (e.g., a cell phone, a personal data assistant (PDA), a personal media player device, a wireless network viewing device, a personal headset device, a dedicated device, or a hybrid device including any of the above functionality). The computing device 300 may also be implemented as a personal computer including both laptop and non-laptop computer configurations.

[0088] Based on the foregoing, it will be appreciated that the specific embodiments of the present disclosure have been described herein for illustrative purposes, but various modifications may be made without departing from the present disclosure. Additionally, many elements of the elements of an embodiment may be added to or replaced with elements of other embodiments and combined with other embodiments. Therefore, except for the appended claims, the present technology is not limited.

Claims

1. A method for granular change detection of documents stored in a distributed storage system having a plurality of servers interconnected with each other via a computer network, the method comprising: receiving, at a first server, data representing a new version of a document stored in the distributed storage system, the received new version of the document having a plurality of attributes that individually have values ​​that describe or identify the document; performing a comparison of the values ​​of respective properties of the document in the received new version with the values ​​of corresponding properties in a previous version of the document in the distributed storage system; Based on the performed comparison, identifying a property whose value in a new version of the document has changed when compared to a similar property in the previous version; as well as Based on identifying the one attribute, the notification is sent, via the computer network, to a computing service that was previously registered to receive notifications regarding changes to the identified one attribute.

2. The method according to claim 1, wherein: Performing the comparison includes: applying a hash function to the value of the one of the attributes to generate a hash value; and A comparison of the generated hash value with another hash value of the one of the attributes of the previous version is performed.

3. The method according to claim 1, wherein: Performing the comparison includes: applying a hash function to a value of at least one of the attributes to generate a hash value; determining whether the generated hash value is different from another hash value of the one of the attributes of the previous version; and In response to determining that the generated hash value is different from another hash value of the one of the attributes of the previous version, indicating that the value of the one of the attributes has changed.

4. The method according to claim 1, wherein: Performing the comparison includes: applying a hash function to said value of said one of said attributes to generate a hash value; determining whether the generated hash value is different from another hash value of the one of the attributes of the previous version; and In response to determining that the generated hash value is different from another hash value of the one of the attributes of the previous version, performing the following operations: indicating that the value of said one of said attributes has changed; and The generated hash value is used to replace the other hash value in the distributed storage system.

5. The method according to claim 1, wherein: Performing the comparison includes: applying a hash function to said value of said one of said attributes to generate a hash value; determining whether the generated hash value is different from another hash value of the one of the attributes of the previous version; and In response to determining that the generated hash value is different from another hash value of the one of the attributes of the previous version, performing the following operations: indicating that the value of said one of said attributes has changed; and The generated hash value replaces the other hash value and is stored in the distributed storage system as metadata of the document.

6. The method according to claim 1, further comprising: After identifying the one of the attributes, the following operations are performed: Based on the list stored at the distributed storage system, the computing services that were previously registered to receive notifications regarding changes to the identified one of the properties are identified.

7. The method according to claim 1, wherein: said one of said attributes being a first attribute; The computing service is a first computing service; The notification is a first notification; as well as After identifying that the second property of the document has changed, a second notification is sent, via the computer network, to a second one or more computing services based on the detected change in the second property of the document without sending a notification to the first computing service.

8. The method according to claim 1, wherein: Based on the configuration file, each of the attributes is associated with a hash function; as well as Performing the comparison includes: applying the hash function to the value of the one of the corresponding attributes to generate a hash value; as well as A comparison of the generated hash value with other corresponding hash values ​​of the attribute of the previous version is performed.

9. The method according to claim 1, further comprising: Based on the received notification, a corresponding user experience is provided to the user of the distributed storage system.

10. A computing device for performing granular change detection on data stored in a distributed storage system, the distributed storage system having a plurality of servers interconnected with each other via a computer network, the computing device comprising: processor; as well as A memory having instructions executable by the processor to cause the computing device to perform a process according to one of claims 1-9.

Images