Dynamic hash function combination for change detection in distributed storage systems

Abstract

Disclosed herein is a technique for dynamic hash function combination for change detection in a distributed storage system. In one embodiment, a method includes: dynamically selecting a hash function for an attribute of a new version of a document, and generating a hash value of the value of the attribute using the selected hash function. The method may then include: determining whether the generated hash value of the attribute of the received new version is different from the hash value of the attribute of the previous version of the document in the distributed storage system. In response to determining that the generated hash value is different from the hash value of the previous version of the document, a notification may be sent to one or more computing services that were previously registered to receive notifications about attribute changes, without sending the notification to other computing services that are not registered to receive the notification.

Description

Background Art

[0001] Distributed storage systems typically include routers, switches, bridges, and other network devices that interconnect a large number of computer servers, network storage devices, and other types of computing devices via wired or wireless network links. A computer server may host one or more virtual machines, containers, or other types of virtualized components to provide various computing and / or storage services to users. For example, a computer server may be configured to provide data storage and retrieval services that allow users to store, edit, retrieve, or perform other data management tasks. Technical Field

[0002] The present disclosure relates to the field of computer programs and systems, and more particularly, to methods and computing devices for dynamic hash function combination for change detection of data. Summary of the invention

[0003] This summary is provided to introduce a selection of design concepts that are further described below in the detailed description in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] Today's distributed storage systems are typically arranged in a layered architecture to provide reliable and scalable storage services to a large number of users. Each layer of the distributed storage system can rely on a corresponding computing system to provide the designed services. For example, a distributed storage system may include a storage layer that is highly scalable and capable of storing large amounts of data. The storage layer may typically include a set of back-end servers that are configured to facilitate storage, retrieval, and / or other data management tasks for users. The set of back-end servers may also provide computing resources for performing various data analysis or analytical tasks. Examples of such analytical tasks include aggregation of document views, detection of modification signals, calculation of trending documents, etc. An example of such a storage layer is Microsoft Outlook 2016, provided by Microsoft Corporation of Redmond, Washington.

[0005] While the storage layer is suitable for performing storage and analysis of large amounts of data, the data structures used in the storage layer may not be suitable for providing random access to individual data items. For example, the storage layer may not access data items in a list in a constant time that is independent of the position of the data item in the list or the size of the list. In addition, scalability of such distributed storage systems is often achieved by splitting or "sharding" the stored data or indexes to such data into partitions. For example, in some distributed storage systems, the main index of a data item in the storage layer may be split into multiple sub-indexes. Instead of referencing a data item, the main index may reference a sub-index, which in turn may reference a data item. In such a distributed storage system, when a user requests a stored data item, multiple operations (e.g., fan-out joins) are performed in the storage layer to resolve such references. Performing multiple operations may result in high latency and computational load to serve user requests.

[0006] A technique for providing fast random access to stored data is to cache a subset of the stored data (sometimes referred to as "high-impact data") in a centralized repository configured to provide a caching service for the high-impact data. For example, a set of analysis servers and / or computing processes (referred to herein as "ingestion processors") running in a storage layer can be configured to push a subset of the stored data as high-impact data to a centralized repository. The high-impact data pushed to the centralized repository may include various categories of stored data that may be of interest to various scenario computing services. In one example, the high-impact data for documents stored in the storage layer cached in the centralized repository may include the document name, document extension (e.g., "txt"), the date / time of the last update, the number of times the document has been viewed, the number of times the document has been modified, the universal resource locator (URL) from which the document can be retrieved, and / or other suitable information related to the document. By caching the high-impact data, users can easily retrieve documents and / or other relevant information about the document from the centralized repository.

[0007] The scene computing service can be configured to retrieve cached high-impact data from a centralized repository and react to changes in the underlying data stored in the storage layer to provide a corresponding user experience. For example, in the document example above, the search index service may be interested in changes in the title, body, appendix, or other suitable types of document content for updating the search index. On the other hand, the document metadata service may be interested in the document's viewer list and the document's last modified timestamp for updating the number of views / edits of the document.

[0008] In some distributed storage systems, once any change is detected in a stored document, a blanket notification is sent to all scene computing services for further processing. Examples of detected changes may include write operations to the attributes of a document. However, such blanket notifications may result in a waste of computing resources in a distributed storage system. First, the transmission of blanket notifications may involve resource-intensive notification calls (e.g., hypertext transfer protocol or "HTTP" requests or remote procedure calls). Second, different scene computing services may be interested in changes in different attributes of a document. For example, in the document example, the search index service may not be interested in changes in the document viewer list or the number of modifications to a document used to update the search index. On the other hand, the document metadata service may not be interested in changes in the document body or title. Therefore, the changes indicated in the blanket notification may apply to some scene computing services, but not to other scene computing services. Although the changes indicated in the blanket notification are not relevant, all scene computing services may consume additional computing, network, input / output, and / or other suitable types of resources to determine whether the changes indicated in the blanket notification are related to individual scene computing services. For example, various types of resources may be consumed to perform additional read operations of high-impact data at a centralized repository. Such read operations may result in high computational loads on the servers hosting the centralized repository, high network bandwidth consumption in the computer network, and / or other negative impacts on the performance and / or operation of the distributed storage system.

[0009] Several embodiments of the disclosed technology can address at least some aspects of the above-mentioned deficiencies by implementing granular change detection of document attributes in a distributed storage system. In some embodiments, the distributed storage system may include a storage layer having one or more back-end servers, one or more centralized storage servers, and one or more scene computing servers operably coupled to each other via a computer network. The centralized repository server can be configured to provide centralized repository and cache services for a subset of the data stored in the storage layer. The scene computing server can be configured to provide various types of computing services, such as the above-mentioned search indexing service and document metadata service, in response to changes in certain attributes of documents stored in the back-end storage server.

[0010] The back-end server can be configured to provide various data storage services to facilitate storage, retrieval, modification, deletion and / or other suitable types of data management operations. The back-end server can also be configured to provide an ingestion processor that is configured to analyze new and / or updated data received and stored in the storage layer. In some implementations, the ingestion processor can be configured to receive a registration or other suitable type of indication from the scene computing service. The registration indicates one or more attributes of the document and changes that the corresponding scene computing service wishes to receive notification of. Such a registration can be stored in the data storage unit as a list, table, or other suitable data structure.

[0011] In some embodiments, the centralized repository may include a change detector configured to perform granular change detection on one or more attributes of data items received at the ingestion processor. Although the change detector is described below as part of the centralized repository, in other embodiments, the change detector may also be configured as part of the ingestion processor, as an independent computing service in a distributed storage system, or as other suitable configurations.

[0012] Using a document as an example, when a new version of a document is received at an ingestion processor, the ingestion processor may extract certain high-impact data from the document and write the extracted high-impact data to a centralized repository. Furthermore, a change detector at the centralized repository (or a component thereof) may be configured to compare various values ​​of attributes of the document in the received new version with various values ​​of attributes in a previous version stored in the centralized repository. For example, the change detector may compare the body and / or title of the document to determine whether a change is detected in the body and / or title. In another example, the change detector may also compare the number of views and / or modifications of the document in the new version with the number of views and / or modifications in the old version.

[0013] In response to detecting a change in at least one of the attributes, the change detector can be configured to provide an indication of the detected change to the ingestion processor. In response, the ingestion processor can be configured to generate a notification and send the notification to those scene computing services that have registered to receive notifications of the detected attribute changes. For example, when a change in the body and / or title of a document is detected, the ingestion processor can be configured to send a notification to a search index service instead of a document metadata service. On the other hand, when a change in the number of views or modifications is detected, the ingestion processor can be configured to send a notification to a document metadata service instead of a search index service. Therefore, the number of notifications sent by the ingestion processor and subsequent read operations used to verify the relevance of the notifications can be reduced, resulting in reduced computational load, reduced network bandwidth consumption, and / or reduced consumption of other resources in the distributed storage system.

[0014] In some embodiments, detecting which one or more attributes have changed may include associating hash functions with the attributes of the document and executing the associated hash functions at the ingestion processor to facilitate determining whether the corresponding attributes have changed. For example, the ingestion processor may be configured to apply the hash functions associated with the respective attributes to the values ​​of the attributes in the received new version of the document to derive a hash value (referred to herein as a "new hash value" or H new ). The ingestion processor can then push the generated new hash value to the centralized repository as part of the high-impact data.

[0015] Upon receiving a new hash value, the change detector at the centralized repository may be configured to compare the new hash value with a previous value of the same attribute of a document stored in the centralized repository (referred to herein as a “previous hash value” or H old ). If the new hash value is different from the previous hash value, the change detector can indicate to the ingestion processor that a change in a document attribute has been detected. If the new hash value matches or substantially matches (e.g., a match greater than 90%) the previous hash value, the change detector can indicate that the corresponding attribute has not changed.

[0016] Executing the hash function in the ingestion processor can take advantage of the high scalability of the storage layer to reduce the risk of computing resources in the storage layer becoming a bottleneck. In addition, by selecting a hash function suitable for each attribute, as described in more detail below, the consumption of computing resources at the storage layer can be further reduced and the derived hash values ​​can have a small data size. The small data size of the generated value can result in low storage overhead in the centralized repository, low network bandwidth consumption, and efficient hash comparison when determining whether an attribute has changed.

[0017] Aspects of the disclosed technology also relate to selecting appropriate hash functions to associate with various attributes of stored data items (e.g., the above-mentioned documents). In some implementations, multiple different hash functions can be statically associated with corresponding attributes of a document. For example, an identity function that returns an input value as an output can be used for the number of times a document has been viewed and the number of times it has been modified. In another example, a xxHash function can be associated with the body of a document.

[0018] In other embodiments, the association of a hash function with an attribute can be dynamic, such as based on the data type and / or size of the value of the attribute. For example, an identity function can be used for an attribute whose value is an integer or a short string (e.g., less than 10 or other suitable number of characters). Therefore, the number of views and the number of modifications to a document can be associated with an identity function. For values ​​that are not integers or short strings, other hash functions are configured to map the data of the attribute value to a hash value of a fixed size that is different from the data size of the value. For example, the Fowler-Noll-Vo (FNV) hash function can be associated with an attribute with a value of four to twenty bytes of data size. The xxHash64 function can be associated with an attribute with a value of a large data size (e.g., greater than 20 bytes). The secure hash heuristic 256 (SHA256) function can be applied to attribute values ​​when conflicts of the same data cannot be tolerated, such as when the number of changes to be detected is higher than a threshold. For example, when almost every change to an attribute needs to be detected, applying SHA256 to the attribute value can produce a 256-bit hash value (or other threshold bit hash value) that achieves a much lower risk of generating collisions.

[0019] In some embodiments, the ingest processor may be configured to use a configuration file that associates a hash function with a range of value sizes for an attribute. For example, attributes with values ​​less than or equal to four bytes may be associated with the identity function, while attributes with values ​​between five and twenty bytes may be associated with the FNV hash function. Attributes with values ​​greater than twenty bytes may be associated with xxHash64. Additionally, attributes that cannot tolerate collisions under any circumstances may be associated with SHA256.

[0020] In other embodiments, the ingestion processor can be configured to present the dynamic selection of a hash function as an optimization operation using a cost function. Different hash functions have different storage space (e.g., output hash value size), computational resource costs (e.g., computational cost of computing a hash value), collision rates, and / or other suitable characteristics. Thus, the optimization operation can be defined as: for an attribute, selecting a hash function to minimize processor cycles and storage overhead while keeping the number of collisions below a threshold and minimizing the total amount of attribute data that must be read to perform a full comparison. Example cost function J( Hi ) can be expressed as follows:

[0021] J( Hi )=W Collision *CR Hi +W CPU *CPU Hi +W Storage *Storage Hi +W Data *DataHi , where CR Hi is the conflict rate, CPU Hi is the processor cycles, Storage Hi is the storage size of the hash value, Data Hi ∈{PropertySize,Storage Hi}, and W is the weight associated with the various corresponding parameters.

[0022] When the distributed storage system is running, the different weights associated with the aforementioned parameters may change. For example, the available system resources (e.g., computing or storage) may change, thereby making computing resources more abundant. Therefore, the weights used for computing resources may be adjusted to cause a different hash function to be selected for the same attribute than before the available system resources changed.

[0023] Additional aspects of the disclosed technology relate to performing change detection by grouping related attributes together and generating a hash value for the group of attributes rather than for each attribute in the group. Without being bound by theory, it is recognized that certain attributes of a data item (e.g., a document) often change together. For example, the number of views often changes with the number of modifications to a document. Therefore, combining attributes into a group and generating a hash value for the group can enable a quick determination of whether any attribute in the group has changed. If no changes to the group are detected, a full comparison of each attribute in the group can be skipped, and metadata can be added to the new version to indicate accordingly. If changes in a group are detected, each attribute in the group can be compared to see which one or more attributes have changed.

[0024] According to additional aspects of the disclosed technology, grouping attributes can also use optimization operations. If the correlation of changes between different attributes is used to group the attributes, a "group correlation threshold" can be set so that the amount of data to be read for attribute and hash comparison is below a size threshold and / or minimized. The correlation of the change can include data indicating the probability (e.g., as a percentage) that the first attribute may change when the second attribute changes, and vice versa. The correlation of the change can be developed via statistical analysis of historical data of the attribute or via other suitable means. In some embodiments, if the value of the attribute has been hashed, a combination operation (e.g., left shift and perform bitwise XOR) can be used to derive a group hash value. However, if the attribute has passed an identity function, an additional hash operation can be performed on the combination of attribute values ​​(referred to as a group value) to derive a group hash value. In other embodiments, multiple groupings of attributes can be used, wherein the attribute can be part of multiple groups.

[0025] A technique for generating attribute groupings may include maintaining a covariance matrix for the attributes. A covariance matrix is ​​a matrix whose element at position i, j is the covariance (i.e., a measure of joint variability) between the i-th and j-th elements of a random vector. The calculation of the covariance matrix can be simplified as follows: if the attribute has changed, the value of the attribute is 1. If the attribute has not changed, the value of the attribute is 0. If the attribute is different from the attribute group P that is already in the cluster, the attribute is considered to be a random vector. Cj Some (or all) of the attributes in <Pi,P Cj >corresponding value is above threshold Tc), a clustering heuristic can be used to construct a grouping by assigning attributes Pi to existing clusters (attribute groups). If the attributes are not relevant enough, i.e., no attribute Pi has a value above threshold Tc in the covariance matrix, a new cluster (i.e., a new attribute group) containing only the attribute can be created. Another grouping heuristic may include determining the "closest" cluster based on a cluster similarity function and assigning the attributes to the closest cluster (i.e., attribute group). Regardless of which heuristic is used to create the attribute grouping, the amount of data to be read and compared in order to detect which attributes have changed can be kept below a threshold and / or minimized.

[0026] In some embodiments, the attribute groupings may be calculated before the distributed storage system starts running, but the attribute groupings may also be calculated periodically in an offline batch mode. After the attribute groupings are determined, a bit field in which each bit corresponds to an attribute may be used to record which attribute has changed. For example, if the corresponding attribute has not changed, the bit may be set to zero. When the corresponding attribute changes, the same bit may be set to 1. In other embodiments, the attribute groupings may be changed in an online manner to better accommodate temporal changes in the data. In other embodiments, a single attribute may be included in multiple different groups. This inclusion may allow changes to be detected at any time. For example, when a change is detected in one group and not in the other, it may be determined that a certain attribute in both groups has not changed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram illustrating a distributed storage system implementing granular change detection according to an embodiment of the disclosed technology.

[0028] Figure 2A-2D is a description of an embodiment according to the disclosed technology, Figure 1 A schematic diagram of some hardware / software components of a distributed storage system for performing granular change detection in .

[0029] Figure 3A-3C is a diagram illustrating an embodiment of the disclosed technology during signature generation. Figure 2A-2DSchematic diagram of some of the hardware / software components of the signature assembly.

[0030] Figure 4 is a description of an embodiment according to the disclosed technology, Figure 2A-2D Schematic diagram of the grouping of properties of certain hardware / software components implementing the signature component.

[0031] Figure 5 is a Venn diagram illustrating one example grouping of attributes in accordance with an embodiment of the disclosed technology.

[0032] Figure 6A-6C is a flow chart illustrating a granularity change detection process in a distributed storage system according to an embodiment of the disclosed technology.

[0033] Figure 7 Is applicable to Figure 1 A computing device that is some component of a distributed storage system. DETAILED DESCRIPTION

[0034] Certain embodiments of systems, devices, components, modules, routines, data structures, and processes for granular change detection in a data center or other suitable distributed storage system are described below. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the relevant art will also understand that the technology may have additional embodiments. It may also be described without reference to the following. Figure 1-Figure 7 The techniques are implemented without describing several details of the embodiments.

[0035] As used herein, the term "distributed storage system" generally refers to an interconnected computer system having routers, switches, bridges, and other network devices that connect a large number of computer servers, network storage devices, and other types of computing devices to each other and / or to an external network (e.g., the Internet) via wired or wireless network links. Individual servers or storage nodes may include one or more persistent storage devices, such as hard disk drives, solid-state drives, or other suitable computer-readable storage media.

[0036] As also used herein, a "document" generally refers to a computer file that contains various types of data and metadata that describe and / or identify the computer file. The various types of data and metadata are referred to herein as "properties" of the document. Example properties may include a title, content body, author, universal resource locator (URL), number of views, number of edits, and / or other suitable information related to the document. Each property may be defined as a parameter containing a corresponding value. For example, an author property may have a value containing a name, such as "Nick Jones". A number of views property may include a value containing an integer, such as "4".

[0037] As used herein, a "version" of a document generally refers to a state or form of a document. A version may be defined by a version number, edit date / time, editor name, saved date / time, and / or other suitable attributes. A "new version" generally refers to a state of a document in which there is at least one change in any aspect of the document's data and / or metadata. For example, a new version of a document may include edits to the document's content or changes to one or more attributes of the document (e.g., number of views). A "previous version" is another state of a document that may be derived directly or indirectly from a previous version.

[0038] "Scenario computing services", also used in this article, generally refer to one or more computing resources provided through a computer network (such as the Internet designed to provide a designed user experience). Example scenario computing services include software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). SaaS is a software distribution technology in which software applications are hosted by a cloud service provider, such as a data center, and accessed by users through a computer network. PaaS generally refers to the delivery of operating systems and related services over a computer network without downloading or installing. IaaS generally refers to outsourced equipment used to support storage, hardware, servers, network equipment, or other components, all of which can be accessed through a computer network.

[0039] Various scene computing services can be configured to provide different user experiences. One example scene computing service can be configured to compile various attributes of a document into a search index. In another example, another scene computing service can be configured to receive a signal indicating that a user viewed a document, and in response to the received signal, provide an electronic message to other users to recommend viewing the same document. In further examples, the scene computing service can be configured to provide other suitable user experiences.

[0040] In some distributed storage systems, once any change is detected in a document, a blanket notification is sent to all scene computing services for further processing. However, such blanket notifications can lead to a waste of computing resources in the distributed storage system. First, the transmission of blanket notifications may involve resource-intensive notification calls (e.g., HTTP requests or remote procedure calls). Second, different scene computing services may be interested in changes in different properties of a document. For example, a search index service may not be interested in changes in a list of document viewers or the number of modifications to a document used to update the search index. On the other hand, a document metadata service may not be interested in changes in the body or title of a document. Therefore, the changes indicated in the blanket notification may apply to some scene computing services but not to other scene computing services.

[0041] Although the changes indicated in the summary notification are not related, all scene computing services may consume additional computing, network, input / output, and / or other suitable types of resources to determine whether the changes indicated in the summary notification are related to the individual scene computing services. For example, various types of resources may be consumed to perform additional read operations. Such read operations may result in high computing loads, high network bandwidth in the computer network, and / or other negative effects on the performance and / or operation of the distributed storage system.

[0042] Several embodiments of the disclosed technology can address at least some aspects of the above-mentioned deficiencies by implementing granular change detection of document properties in a distributed storage system. In certain embodiments, a change detector can be implemented in a distributed storage system to determine which one or more properties have changed in a new version of a document when compared to a previous version of the same document. Based on the determined one or more properties, a notification can be selectively sent to a scene computing service that was previously registered to receive notifications of changes to one or more properties. Thus, costly HTTP requests and / or remote procedure calls to scene computing services that are not interested in the detected property changes can be avoided. In addition, additional computing resources for determining whether the changes indicated in the notification are actually related to the scene computing service can also be avoided. Therefore, when compared to using blanket notifications, the computing load, network bandwidth, and / or consumption of other computing resources of the distributed storage system can be reduced, as described below with reference to Figure 1-Figure 7 Described in more detail.

[0043] Figure 1 1 is a schematic diagram illustrating a computing environment 100 for implementing granular change detection according to an embodiment of the disclosed technology. Figure 1 As shown, the computing environment 100 may include one or more client devices 102 corresponding to a user 101 and a distributed storage system 103 interconnected to the client devices 102 via a computer network (not shown) such as the Internet. Figure 1 Specific components of the computing environment 100 are shown in the figure, but in other embodiments, the computing environment 100 may also include additional and / or different components or arrangements. For example, in some embodiments, the computing environment 100 may also include additional network storage devices, network devices, hosts and / or other suitable components (not shown) in other suitable configurations.

[0044] Each of the client devices 102 may include a computing device for facilitating user 101 to access a cloud storage service provided by the distributed storage system 103 via a computer network. In one embodiment, each of the client devices 102 may include a desktop computer. In other embodiments, the client device 102 may also include a laptop computer, a tablet computer, a smart phone, or other suitable computing device. Figure 1 Two users 101 and 101 ′ are shown for illustration purposes, but in other embodiments, the computing environment 100 may facilitate other suitable numbers of users 101 to access cloud storage or other types of computing services provided by the distributed storage system 103 in the computing environment 100 .

[0045] like Figure 1 As shown, the distributed storage system 103 may include one or more front-end servers 104, one or more back-end servers 105, a centralized repository 106, and one or more scene servers 108 operably coupled via a computer network 109. The back-end server 105 may be operably coupled to a network storage unit 112 configured to store user data. In the illustrated example, the user data stored at the network storage unit 112 is shown as a document 110. In other examples, the user data may also include image files, video files, or other suitable types of computer files. Although in the example of FIG. Figure 1 Specific components of the distributed storage system 103 are shown in FIG. 1 , but in other implementations, the distributed storage system 103 may also include additional and / or different components.

[0046] The computer network 109 may include any suitable type of network. For example, in one embodiment, the computer network 109 may include an Ethernet or Fast Ethernet network with routers, switches, load balancers, firewalls, and / or other suitable network components that implement the RDMA over Converged Ethernet (RoCE) protocol. In other embodiments, the computer network 109 may also include an InfiniBand network with corresponding network components. In other embodiments, the computer network 109 may also include a combination of the above and / or other suitable types of computer networks.

[0047] The front-end server 104 may be configured to interact with the user 101 via the client device 102. For example, the front-end server 104 may be configured to provide a user interface (not shown) on the client device 102 to allow the corresponding user 101 to upload, modify, retrieve or perform other suitable operations on the document 110 stored in the network storage unit 112. In another example, the front-end server 104 may also be configured to provide a user portal (not shown) that allows the user 101 to set retention, privacy, access or other suitable types of policies to be implemented on the document 110 in the network storage unit 112. In some embodiments, the front-end server 104 may include a network server, a security server and / or other suitable types of servers. In other embodiments, the front-end server 104 may be omitted from the distributed storage system 103 and instead provided by an external computing system, such as a cloud computing system (not shown).

[0048] The backend server 105 may be configured to provide various data storage services to facilitate storage, retrieval, modification, deletion, and / or other suitable types of data management operations on the documents 110 in the network storage unit 112. For example, the backend server 105 may be configured to store the documents 110 and / or indexes of the documents 110 in the shards. The backend server 105 may also be configured to provide facilities for performing fan-out joins of data from various shards in response to appropriate queries from, for example, the user 101. In further examples, the backend server 105 may also be configured to provide facilities for performing duplication reduction, data compression, access control, and / or other suitable operations.

[0049] The backend server 105 may also be configured to provide an ingestion processor 107 ( Figure 2A-2D ), the ingestion processor is configured to analyze new and / or updated data of the document 110 received at and stored in the network storage unit 112. In some implementations, the ingestion processor 107 can be configured to receive a registration or other suitable type of indication from the scene server 108 and / or the corresponding scene computing service. The registration can indicate changes in one or more attributes of the document 110, the corresponding scene server or the scene computing service, respectively, that they wish to receive notifications of. Such a registration can be provided as a notification list 128 ( Figure 2D The data may be maintained in other suitable data structures such as those shown), tables, or other suitable data storage units.

[0050] Although the backend server 105 may be suitable for performing storage and analysis of large amounts of data, the data structures used in the backend server 105 and / or the network storage unit 112 may not be suitable for providing random access to individual data items (e.g., documents 110). For example, the backend server 105 may not access a document 110 in a list at a constant time that is independent of the position of the document 110 in the list or the size of the list. In addition, when a user requests a stored document, the backend server 105 may perform multiple operations (e.g., fan-out joins) to resolve indirect references. Performing multiple operations may result in high latency and computational load to service the user request.

[0051] Centralized repository 106 may be configured to provide fast random access to documents 110 stored in network storage unit 112 by providing a caching service to a subset of documents 110 cached in data storage unit 114 (sometimes referred to as "high impact data" in the example of FIG. 1 ). Figure 1In some embodiments, the ingestion processor 107 running in the backend server 105 can be configured to push a subset of the stored data of the document 110 as data 111 to the centralized repository 106 for storage in the data storage unit 114.

[0052] The data 111 pushed to the centralized repository 106 may include various categories of stored data that may be of interest to various scene servers 108 and / or scene computing services. In one example, the data 111 cached in the centralized repository 106 for the document 110 stored in the network storage unit 112 may include the document name, document extension (e.g., "txt"), the date / time of the last update, the number of times the document has been viewed, the number of times the document has been modified, the universal resource locator (URL) from which the document can be retrieved, and / or other suitable information related to the document. By caching the data 111, the user 101 can easily retrieve the document information and / or other related information from the centralized repository 106.

[0053] The scene server 108 may be configured to retrieve cached data 111 from the centralized repository 106 and react to changes in the underlying data of the document 110 stored in the network storage unit 112 to provide a corresponding user experience. For example, the search index service may be interested in changes to the title, body, appendix, or other suitable type of document 110 content for updating the search index (not shown). On the other hand, the document metadata service may be interested in the view count or viewer list of the document 110 for updating the number of views / edits of the document 110 and the timestamp of the last modification of the document 110. In response to the change in the view count, the document metadata service may be configured to send a message 118 to, for example, user 101' to notify user 101' that another user 101 viewed and / or edited the document 110. Although the scene server 108 is Figure 1 Although illustrated as a physical computing device, in some implementations, the functionality of the scene server 108 may be provided as a corresponding scene computing service by a suitable server (e.g., the front-end server 104) executing suitable instructions.

[0054] In some distributed storage systems, once any change is detected in a stored document 110, a change notification 116 is sent to all scene servers 108 for further processing. Examples of detected changes may include write operations to attributes of the document 110. However, such blanket notifications may result in a waste of computing resources in the distributed storage system 103. First, the transmission of the change notification 116 may involve resource-intensive notification calls (e.g., Hypertext Transfer Protocol or "HTTP" requests or remote procedure calls).

[0055] Secondly, different scene servers or scene computing services may be interested in changes in different properties of the document 110. For example, the search index service may not be interested in the viewer list of the document 110 used to update the search index or the change in the number of modifications to the document 110. On the other hand, the document metadata service may not be interested in changes in the body or title of the document 110. Therefore, the changes indicated in the change notification 116 may be applicable to some scene computing services, but not to other scene computing services. Although the changes indicated in the change notification 116 are not related, all scene computing services may consume additional computing, network, input / output and / or other suitable types of resources to determine whether the changes indicated in the change notification 116 are related to individual scene computing services. For example, various types of resources can be consumed to perform additional read operations of the data 111 at the centralized repository 106. Such read operations can result in high computing loads on the servers hosting the centralized repository 106, high network bandwidth in the computer network 109, and / or other negative effects on the performance and / or operation of the distributed storage system 103.

[0056] Several embodiments of the disclosed technology can address at least some aspects of the above-mentioned deficiencies by implementing granular change detection of document properties in a distributed storage system 103. In some embodiments, a change detector 122 can be implemented in a centralized repository 106 to determine which one or more properties have changed in a new version of a document 110 when compared to a previous version of the same document 110. Based on the determined one or more properties, a change notification 116 can be selectively sent to a scene server 108 that was previously registered to receive notifications of changes to one or more properties. Thus, costly HTTP requests and / or remote procedure calls to scene computing services that are not interested in the detected property changes can be avoided. In addition, additional computing resources for determining whether the changes indicated in the change notification 116 are actually relevant to the scene server 108 can also be avoided. Therefore, when compared to using blanket notifications, the computing load, network bandwidth, and / or consumption of other computing resources of the distributed storage system 103 can be reduced, as described below with reference to Figure 2A-2D Described in more detail.

[0057] Figure 2A-2D is a description of an embodiment according to the disclosed technology, Figure 1 A schematic diagram of certain hardware / software components in a distributed storage system 103 for performing granular change detection. Figure 2A-2D In other figures of this paper, the Figure 1 Certain components of the computing environment 100 and the distributed storage system 103. For example, Figure 2A-2DIn the example, for ease of explanation, the front-end server 104 and the computer network 109 are omitted from the distributed storage system 103. Figure 2A-2D In other figures of this article, each software component, object, class, module and routine can be a computer program, process or process written as source code in C, C++, C#, Java and / or other suitable programming languages. Components can include, but are not limited to, one or more modules, objects, classes, routines, attributes, processes, threads, executable files, libraries or other components. Components can be in source code form or binary form. Components can include aspects of source code before compilation (e.g., classes, attributes, processes, routines), compiled binary units (e.g., libraries, executable files) or artifacts (e.g., objects, processes, threads) that are instantiated and used at runtime.

[0058] Components within the system may take different forms within the system. As an example, a system including a first component, a second component, and a third component may include, but is not limited to, a system having a first component as a property in source code, a second component as a binary compiled library, and a third component as a thread created at runtime. A computer program, procedure, or process may be compiled into an object, intermediate code, or machine code and presented for execution by one or more processors of a personal computer, a network server, a laptop, a smartphone, and / or other suitable computing device.

[0059] Likewise, a component may include a hardware circuit. One of ordinary skill in the art will recognize that hardware may be viewed as petrified software and software may be viewed as liquefied hardware. As just one example, the software instructions in a component may be burned into a programmable logic array circuit, or may be designed as a hardware circuit with an appropriate integrated circuit. Likewise, hardware may be emulated by software. Various implementations of source code, intermediate code, and / or object code and associated data may be stored in a computer memory, including a read-only memory, a random access memory, a disk storage medium, an optical storage medium, a flash memory device, and / or other suitable computer-readable storage medium other than the propagated signal.

[0060] like Figure 2A As shown, the back-end server 105 may include a Figure 1 ) from the client device 102 of the user 101. The document 110 may include a plurality of attributes 120. In the illustrated embodiment, the ingestion processor 107 may include an interface component 130, a signature component 132, and a notification component 134 that are operably coupled to each other. In other embodiments, the ingestion processor 107 may also include network, security, database, or other suitable types of components.

[0061] The interface component 130 may be configured to receive a new version of the document 110 and provide the received document 110 to the signature component 132 for further processing. The interface component 130 may also be configured to store, retrieve, or perform other suitable file operations on the document 110 in the network storage unit 112. For example, in the illustrated embodiment, the interface component 130 may be configured to store a copy of the received new version of the document 110 in the network storage unit 112 as a separate file from the previous version, or to replace the previous version of the document 110.

[0062] The signature component 132 may be configured to generate a signature 121 corresponding to one or more of the attributes 120. In some embodiments, the signature 121 of the attribute 120 may include a hash value of the attribute 120. For example, the signature 121 of the content body of the document 110 may include a random alphanumeric string of a preset length. In other embodiments, the signature of the attribute 120 may include a value of the attribute 120. For example, the value of the view count attribute of the document 110 may include the integer "4". In other examples, the signature 121 may include other suitable values ​​derived from and / or corresponding to various attributes 120 of the document 110.

[0063] like Figure 2A As shown, after generating the signature 121, the signature component 132 can instruct the interface component 130 to send the data 111 extracted from the new version of the document 110 together with the signature 121 to the centralized repository 106 for granular change detection. Although the embodiment of granular change detection is described herein as being implemented in the centralized repository 106, in other embodiments, such granular change detection can also be implemented on one or more of the backend server 105 or other suitable servers (not shown) of the distributed storage system 103.

[0064] like Figure 2A As shown, the centralized repository 106 may include a change detector 122 and a change indicator 124 that are operably coupled to each other. The change detector 122 may be configured to determine which one or more of the detected attributes have changed. In some embodiments, the change detector 122 may be configured to compare the individual signatures 121 of the new version of the document 110 with those of the previous version stored in the data storage unit 114. In response to determining that the signature 121 of the new version is the same or substantially the same (e.g., the similarity exceeds 90%), the change detector 122 may be configured to indicate that the corresponding attribute 120 of the document 110 has not changed. Otherwise, the change detector 122 may be configured to indicate that the corresponding attribute 120 has changed. Figure 2BAs shown, after iterating through all signatures 121 ( Figure 2A ), the change indicator 124 may be configured to send a list of attribute IDs 123 identifying a list of attributes 120 that have changed to the ingest processor 107 at the backend server 105.

[0065] In response to receiving the attribute ID 123, the notification component 134 of the ingest processor 107 can be configured to retrieve the notification list 128 from the network storage unit 112. The notification list 128 can include data indicating which one or more of the scene servers 108 (and / or corresponding scene computing services) are interested in changes to a particular attribute 120 of the document 110. The notification list 128 can be generated by receiving registration data (not shown) from the scene servers 108, by assigning default values, or via other suitable techniques. Based on the retrieved notification list 128, the notification component 134 can be configured to determine that one or more scene servers 108 received the change notification 125 based on the attribute ID 123. In the illustrated example, the first scene server 108a can be indicated in the notification list 128 as being interested in changes to the attribute 120 identified by one of the attribute IDs 123. Therefore, the change notification 116 is only sent to the first scene server 108a, and not to the second and third scene servers 108b and 108c.

[0066] like Figure 2C As shown, upon receiving the change notification 116, the first scene server 108a may be configured to provide a message 118 to the user 101' to, for example, notify the user 101' of the detected change in the attribute 120 of the document 110, or to provide other suitable user experiences. Figure 2D As shown, when the user 101 provides another version of the document 110' with different changes to one or more of the attributes 120', the notification component 134 can be configured to: when a change in a different attribute 120' is detected, as indicated by the attribute ID 123', provide a change notification 116' to the third scene server 108c instead of the first or second scene servers 108a and 108b. Therefore, resource-intensive notification calls to one or more of the scene servers 108 and additional read operations by such one or more scene servers 108 can be avoided.

[0067] Figure 3A-3C is a diagram illustrating the steps during signature generation according to an embodiment of the disclosed technology. Figure 2A-2D A schematic diagram of some of the hardware / software components of the signature component 132. Figure 3AAs shown, the signature component 132 may include a function selector 142 and a signature generator 146 operatively coupled to each other. The function selector 142 may be configured to select a function for generating the signature 121 based on the value of the attribute 120. As described in more detail below, the function selector 142 may be configured to select the function in various ways. The signature generator 146 may be configured to apply the selected function (e.g., a hash function) to the value of the attribute 120 to generate the signature 121.

[0068] Aspects of the disclosed technology involve selecting appropriate functions associated with various attributes 120 of a document 110 in order to transfer the attributes 120 to the backend server 104. Figure 1 ) is kept below a threshold or even minimized. In some embodiments, Figure 3A As shown, a plurality of different hash functions may be statically assigned and associated with corresponding attributes 120 of a document 110, as shown in a function list 137 stored in the network storage unit 112. For example, an identity function that returns an input value as an output may be used for the number of times a document has been viewed or modified, since the cost of computing hash values ​​for these values ​​may outweigh the savings in computing resources used to compare the values. In another example, the xxHash function may be associated with the body of a document 110, since the body may include a large amount of data.

[0069] In other embodiments, for example, based on the data type and / or size of the value of the attribute 120 or other suitable function criteria 138, such as Figure 3B As shown, the association of the hash function with the attribute 120 can be dynamic. For example, the function selector 142 can be configured to determine the data type and / or size of the value of the attribute 120, and assign a value to the identity function for the attribute 120 according to the function standard 138, the value being an integer or a short string (e.g., less than ten or other suitable number of characters). Therefore, the number of views and the number of modifications to the document can be associated with the identity function. In other examples, the function selector 142 can be configured to assign the Fowler-Noll-Vo (FNV) hash function to the attribute 120 with a value of 4 to 20 bytes of data size according to the function standard 138. In another example, the function selector 142 can also be configured to assign the xxHash64 function to the attribute 120 with a value of a large data size (e.g., greater than 20 bytes) according to the function standard 138. In another example, the function selector 142 may be configured to assign a secure hash heuristic 256 (SHA256) function to attributes 120 having values ​​that may produce hash value collisions, and when such collisions cannot be tolerated.

[0070] In other embodiments, Figure 3CAs shown, the function selector 142 can be configured to use the cost function 139 to present the dynamic selection of the hash function as an optimization operation. Different hash functions can have different storage space (e.g., output hash value size), computing resource cost (e.g., cost of computing hash values), conflict rate, and / or other suitable characteristics. Therefore, the optimization operation can be defined as: for the attribute 120, the hash function is selected to minimize the processor cycles and storage overhead while keeping the number of conflicts below a threshold and minimizing the total amount of attribute data (Data) that must be read for a complete comparison. The example cost function J(Hi) can be expressed as follows: J( Hi )=W Collision *CR Hi +W CPU *CPU Hi +W Storage *Storage Hi +W Data *Data Hi , where CR Hi is the collision rate, CPU Hi is the processor cycles, Storage Hi is the storage size of the hash value, Data Hi ∈{PropertySize,Storage Hi} and W is the weight associated with the various corresponding parameters.

[0071] When the distributed storage system 103 is running, the different weights associated with the aforementioned parameters may change. For example, the available system resources may change, thereby making computing resources abundant. Therefore, the weights used for computing resources may be adjusted to cause a different hash function to be selected for the same attribute 120 than before the available system resources changed.

[0072] Figure 4 is a description of an embodiment according to the disclosed technology, Figure 2A-2D Schematic diagram of the grouping of properties implemented by some hardware / software components of the signature component. Figure 4 As shown, the signature component 132 may also include a grouping component 140, which is configured to: group some of the attributes 120 of the document 110 into a set, generate a signature 121 of the set, and allow the centralized repository 106 ( Figure 2A ) at the change detector 122 ( Figure 2A ) efficiently determine which one or more of the attributes 120 have changed.

[0073] In some embodiments, the grouping component 140 can be configured to group related attributes 120 together and generate the signature 121 for a set of attributes 120 rather than for each attribute 120 in the set. Without being bound by theory, it is recognized that certain attributes 120 of the document 110 often change together. For example, the number of views often changes with the number of modifications to the document 110. Therefore, combining the attributes 120 into groups or sets and generating hash values ​​for the set can enable a quick determination of whether any attribute 120 in the set has changed. For example, if no changes in the set are detected, the change detector 122 can skip a full comparison of each attribute 120 in the set. If a change in the set is detected, each attribute 120 in the set can be compared to see which one or more of the attributes 120 has changed based on the function list 137, the function criteria 138, or the cost function 139, as described above for Figure 3A-3C described.

[0074] According to additional aspects of the disclosed technology, grouping attributes 120 may also use optimization operations. If varying correlations between different attributes 120 are used to group attributes 120, a "grouping correlation threshold" may be set so that the amount of data to be read for the attributes 120 and hash comparisons remains below a size threshold and / or is minimized. In some embodiments, if the values ​​of the attributes 120 have already been hashed, a combination operation (e.g., left shift and bitwise XOR) may be used to generate a group hash value as the signature 121 for the group. However, if the attributes 120 have already passed an identity function, additional hash operations may be performed on the combination of the values ​​of the attributes to generate a group hash value. In other embodiments, multiple groupings of attributes 120 may be used, where the attributes 120 may be part of multiple groups or sets.

[0075] One technique for generating attribute groupings may include maintaining a covariance matrix (not shown) for the attributes 120. The calculation of the covariance matrix may be simplified such that if the attribute 120 has changed, the value of the attribute 120 is 1. If the attribute 120 has not changed, the value of the attribute 120 is 0. If the attribute 120 is not identical to an attribute group P already in the cluster, the covariance matrix may be calculated. Cj Some (or all) of the attributes 120 in <Pi,P Cj>corresponding value is above threshold Tc), a clustering heuristic can be used to construct a grouping by assigning attributes 120Pi to existing clusters (attribute groups). If attribute 120 is not relevant enough, i.e., no attribute Pi value in the covariance matrix is ​​above threshold Tc, a new cluster (i.e., a new attribute group) containing only this attribute can be created. Another grouping heuristic may include determining the "closest" cluster based on a cluster similarity function and assigning attribute 120 to the closest cluster (i.e., attribute group). Regardless of which heuristic is used to create attribute groupings, the amount of data to be read and compared in order to detect which attributes 120 have changed can be kept below a threshold and / or minimized.

[0076] In some embodiments, the attribute groupings may be calculated before the distributed storage system 103 begins to become operational, but the attribute groupings may also be calculated periodically in an offline batch process or other suitable processing manner. After the attribute groupings are determined, a bit field in which each bit corresponds to an attribute 120 may be used to record which attribute 120 in the group has changed. For example, if the corresponding attribute 120 has not changed, the bit may be set to zero. When the corresponding attribute has changed, the same bit may be set to 1. In other embodiments, the attribute groupings may be changed in an online manner to better accommodate temporal changes in data.

[0077] In other embodiments, a single attribute 120 may be included in multiple different attribute groups. Such inclusion may allow for ready detection of changes. For example, Figure 5 As shown, the first group 150a may include four attributes 120 represented as circles, and the second group 150b may also include four attributes 120'. The first and second groups 150a and 150b share common attributes 120". Therefore, when a change is detected in the first group 150a but no change is detected in the second group 150b, it can be determined that the shared attributes 120" in both the first group 150a and the second group 150b have not changed.

[0078] Figure 6A-6C is a flow chart illustrating a process for granularity change detection in a distributed storage system according to an embodiment of the disclosed technology. Figure 1 Embodiments of the processes are described in the context of computing environment 100, but in other embodiments, the processes may also be implemented in other computing environments having additional and / or different components.

[0079] like Fig. 6A As shown, process 200 may include receiving a new version of a document at stage 202. In some embodiments, the new version of a document may have a file stored in distributed storage system 103 ( Figure 1) in the corresponding previous version. In other embodiments, the previous version of the document may not exist in the distributed storage system 103. Then, the process 200 may include: comparing various attributes of the received new version of the document at stage 204. Various techniques for comparing attributes may be applied. Figure 2A-2D and referenced below Figure 6B Certain example techniques are described. Process 200 may also include identifying, at stage 206, properties whose values ​​have changed in the received new version of the document. Upon identifying properties whose values ​​have changed, process 200 may proceed to send notifications to a scene server or scene computing service that was previously registered to receive notifications about property changes.

[0080] Figure 6B This section describes an example operation that compares various attributes of a document. Figure 6B As shown, the operations may include comparing signatures (e.g., hash values) of attribute values ​​210, which includes identifying the values ​​of the respective attributes at stage 212. The operations may optionally include selecting a function for each attribute to generate a signature at stage 214. Figure 3A-3C Various example techniques for selecting a function are described. Operations may then include applying the selected function to the values ​​of the corresponding attributes to generate a signature at stage 216. Based on the generated signature, operations may include detecting changes in one or more attributes at stage 218.

[0081] Figure 6C This section describes an example operation that detects changes in properties by grouping them into sets. Figure 6C As shown, the operation may include: grouping the attributes into sets at stage 222. Figure 4 Example techniques for grouping attributes are described. Operations may then include generating a group signature for each set at stage 224. For each set, operations may include a decision stage 224 for determining whether a change in the set has been detected based on the group signature. In response to determining that a change in the set has not been detected, operations may include marking all attributes in the set as unchanged. In response to determining that a change in the set has been detected, operations may include determining, at stage 228, whether a change in the set has been detected by performing a decision such as described above with reference to FIG. Figure 6B Describes those operations that detect changes to one or more attributes in a collection.

[0082] Figure 7 Is applicable to Figure 1 The computing device 300 may be adapted to include certain components of the computing environment 100. For example, the computing device 300 may be adapted to include certain components of the computing environment 100. Figure 1The computing device 300 includes a client device 102, a front-end server 104, a back-end server 105, a centralized repository 106, and a scene server 108. In a very basic configuration 302, the computing device 300 may include one or more processors 304 and a system memory 306. A memory bus 308 may be used to communicate between the processor 304 and the system memory 306.

[0083] Depending on the desired configuration, the processor 304 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 304 may include one or more levels of cache (e.g., a first-level cache 310 and a second-level cache 312), a processor core 314, and registers 316. An example processor core 314 may include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 318 may also be used with the processor 304, or in some implementations, the memory controller 318 may be an internal part of the processor 304.

[0084] Depending on the desired configuration, system memory 306 may be of any type, including but not limited to volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, flash memory, etc.), or any combination thereof. System memory 306 may include an operating system 320, one or more applications 322, and program data 324. The basic configuration 302 described in Figure 7 are shown by those components within the inner dashed line.

[0085] The computing device 300 may have additional features or functions, as well as additional interfaces for facilitating communication between the basic configuration 302 and any other devices and interfaces. For example, a bus / interface controller 330 may be used to facilitate communication between the basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. The data storage device 332 may be a removable storage device 336, a non-removable storage device 338, or a combination thereof. Examples of removable storage and non-removable storage devices include magnetic disk devices (such as floppy disk drives and hard disk drives (HDD)), optical disk drives (such as compact disk (CD) drives or digital versatile disk (DVD) drives), solid state drives (SSDs), and tape drives, to name a few. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The term "computer-readable storage medium" or "computer-readable storage device" does not include propagated signals and communication media.

[0086] System memory 306, removable storage device 336, and non-removable storage device 338 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 300. Any such computer-readable storage media can be part of computing device 300. The term "computer-readable storage media" does not include propagated signals and communication media.

[0087] The computing device 300 may also include an interface bus 340 for facilitating communication from various interface devices (e.g., output devices 342, peripheral interfaces 344, and communication devices 346) to the basic configuration 302 via the bus / interface controller 330. Example output devices 342 include a graphics processing unit 348 and an audio processing unit 350, which may be configured to communicate with various external devices such as a display or speakers via one or more A / V ports 352. Example peripheral interfaces 344 include a serial interface controller 354 or a parallel interface controller 356, which may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 358. Example communication devices 346 include a network controller 360, which may be arranged to facilitate communication with one or more other computing devices 362 over a network communication link via one or more communication ports 364.

[0088] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules, or other data in a modulated data signal (such as a carrier waveform) or other transmission mechanisms and can include any information transfer medium. A "modulated data signal" can be a signal having one or more characteristics in its characteristic set or changing in a manner to encode the information in the signal. By way of example and not limitation, communication media can include wired media such as a wired network or a direct line connection, and wireless media such as sound, radio frequency (RF), microwaves, infrared (IR) and other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0089] The computing device 300 may be implemented as part of a small portable (or mobile) electronic device such as a cell phone, a personal data assistant (PDA), a personal media player device, a wireless network viewing device, a personal headphone device, a dedicated device, or a hybrid device including any of the above functions. The computing device 300 may also be implemented as a personal computer including both laptop and non-laptop computer configurations.

[0090] As can be understood from the above, specific embodiments of the present disclosure have been described herein for illustrative purposes, but various modifications may be made without departing from the present disclosure. In addition, many elements of an embodiment may be added to or replace the elements of other embodiments and combined with other embodiments. Therefore, except for the appended claims, the present technology is not limited.

Claims

1. A method for dynamic hash function combination for change detection of a document stored in a distributed storage system having a plurality of servers interconnected to each other via a computer network, the method comprising: receiving, at one of the servers, data representing a new version of a document stored in the distributed storage system, the received new version of the document having attributes including values ​​that describe or identify the document; as well as Based on receiving the data representing the new version of the document: a hash function that identifies the property of the new version of the document based at least in part on one or more of a data size of the value of the property in the new version of the document or a data type of the value of the property in the new version of the document; generating a hash value of the value of the property in the new version of the document using the identified hash function; comparing the generated hash value of the property in the new version of the document with a corresponding hash value of the property from a previous version of the document in the distributed storage system; as well as Based on detecting that the generated hash value of the property in the new version of the document is different from the corresponding hash value of the property in the previous version of the document, a notification is sent via the computer network only to one or more computing services that were previously registered to receive notifications about changes to the property, and the notification is not sent to other computing services that are not registered to receive the notification.

2. The method according to claim 1, wherein: Identifying the hash function comprises: When the value of the attribute is an integer or a string having fewer than a threshold number of characters, an identity function is selected as the hash function, wherein the identity function is configured to return an input value as an output.

3. The method according to claim 1, wherein: Identifying the hash function comprises: When the value of the attribute is an integer or a string having fewer than a threshold number of characters, selecting an identity function as the hash function, wherein the identity function is configured to return an input value as an output; and When the value of the attribute is not an integer or a string having fewer than a threshold number of characters, a hash function configured to map data of the value of the data size to a hash value of a fixed size different than the data size of the value is selected.

4. The method according to claim 1, wherein: Identifying the hash function comprises: determining whether a quantity of said attributes in said comparison is above a threshold; and In response to determining that the number of the attributes in the comparison is above the threshold, selecting a hash function configured to generate the hash value having a number of bits that is no less than a threshold number of bits.

5. The method according to claim 1, wherein: Identifying the hash function comprises: identifying the data size of the attribute of the new version of the document; and Based on the identified data size of the attribute, the hash function is determined from a configuration file containing records of ranges of data sizes and corresponding hash functions.

6. The method according to claim 1, wherein: Identifying the hash function comprises: The hash function is selected by minimizing a cost function based on one or more of a size of the hash value, a computational cost of calculating the hash value, and a collision rate of the hash value.

7. The method according to claim 1, wherein: Identifying the hash function comprises: A cost function is used to select a hash function that minimizes processor cycles and storage size of the hash value while keeping the number of collisions below a threshold and minimizing the amount of data of the attribute read for comparisons performed.

8. The method according to claim 1, wherein: Identifying the hash function comprises: A cost function is used to select a hash function that minimizes processor cycles and storage size of the hash value while keeping the number of collisions below a threshold and minimizing the amount of data of the attribute read for the comparison performed, as follows: J( Hi )=W Collision *CR Hi +W CPU *CPU Hi +W Storage *Storage Hi +W Data *Data Hi , Among them, CR Hi is the conflict rate, CPU Hi is the processor cycles, Storage Hi is the storage size of the hash value, Data Hi ∈{attribute size, Storage Hi }, and W is the weight associated with the various corresponding parameters.

9. The method according to claim 8, further comprising: One or more of the weights are adjusted in response to changes in availability of computing or storage resources in the distributed storage system.

10. A computing device for dynamic hash function combination for change detection of data stored in a distributed storage system having a plurality of servers interconnected to each other via a computer network, the computing device comprising: processor; as well as A memory having instructions executable by the processor to cause the computing device to perform the method according to one of claims 1-9.

Images