Storage data management method and device, storage medium and electronic equipment

Abstract

The invention discloses a storage data management method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a plurality of pieces of storage data to be accessed; performing hash fragmentation on the plurality of storage data through the first hash function to obtain at least two data sets; performing hash grouping on the at least two data sets through a second hash function to obtain at least two set groups; a target number of service management objects are allocated to the at least two set groups to manage the multiple pieces of storage data, the number of the set groups in the at least two set groups is determined according to the target number, and the service management objects are management objects responsible for processing access requests of the storage data and providing feedback services. The technical problem that the management efficiency of the stored data is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device, storage medium, and electronic device for managing stored data. Background Art

[0002] In storage data management scenarios, service management objects play a crucial role, responsible for processing and managing stored data. However, when data access volume surges, a single service management object often struggles to cope with the immense load. In such situations, the service management object may experience abnormal conditions such as downtime due to overload, significantly reducing the efficiency of storage data management. Consequently, low storage data management efficiency exists.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, storage medium, and electronic device for managing stored data, so as to at least solve the technical problem of low management efficiency of stored data.

[0005] According to one aspect of an embodiment of the present application, a method for managing stored data is provided, comprising: obtaining multiple stored data to be accessed; performing hash sharding on the multiple stored data using a first hash function to obtain at least two data sets, wherein the first hash function is used to map each stored data in the multiple stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the stored data in the same data set in the at least two data sets are in the same hash value range; performing hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to the data sets in the same set group in the at least two set groups are in the same hash value range; and allocating a target number of service management objects to the at least two set groups to manage the multiple stored data, wherein the number of set groups in the at least two set groups is determined based on the target number, and the service management objects are management objects responsible for processing access requests for the stored data and providing feedback services.

[0006] According to another aspect of an embodiment of the present application, a storage data management device is also provided, including: a first acquisition unit, used to acquire multiple storage data to be accessed; a sharding unit, used to hash the above-mentioned multiple storage data through a first hash function to obtain at least two data sets, wherein the above-mentioned first hash function is used to map each storage data in the above-mentioned multiple storage data to a corresponding first hash value, different data sets in the above-mentioned at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the storage data in the same data set in the above-mentioned at least two data sets are in the same hash value range; a grouping unit, used to perform grouping on the above-mentioned at least two data sets through a second hash function. The rows are hashed and grouped to obtain at least two collection groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different collection groups in the at least two collection groups correspond to different second hash values, and the second hash values ​​corresponding to the data sets in the same collection group in the at least two collection groups are in the same hash value range; a management unit is used to allocate a target number of service management objects to the at least two collection groups to manage the multiple storage data, wherein the number of collection groups in the at least two collection groups is determined based on the target number, and the service management objects are management objects responsible for processing access requests to the storage data and providing feedback services.

[0007] As an optional scheme, the above-mentioned grouping unit includes: a determination module, used to determine the above-mentioned number of collection groups based on the above-mentioned target number, wherein each data set in the above-mentioned at least two data sets is set to allocate at least N of the above-mentioned service management objects, and the above-mentioned target number is greater than or equal to the product of N and the above-mentioned number of collection groups, and N is an integer greater than 1; a mapping module, used to map each data set in the above-mentioned at least two data sets to the above-mentioned corresponding second hash value through the above-mentioned second hash function; an allocation module, used to allocate each data set in the above-mentioned at least two data sets to the collection groups of the above-mentioned number of collection groups according to the hash value range of the above-mentioned corresponding second hash value, to obtain at least two collection groups.

[0008] As an optional solution, the above-mentioned device also includes: a routing module, which is used to allocate a target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple stored data. When a first data access request triggered by the first target data is obtained, a random routing method is used to determine the first management object from the multiple service management objects allocated to the target collection group to which the above-mentioned first target data belongs. The above-mentioned first management object is responsible for processing the above-mentioned first data access request and providing feedback services corresponding to the above-mentioned first target data.

[0009] As an optional solution, the above-mentioned device also includes: a first allocation unit, which is used to allocate a corresponding Bloom filter to each data set in the above-mentioned at least two data sets after hash sharding the above-mentioned multiple stored data through the first hash function to obtain at least two data sets, wherein the above-mentioned Bloom filter is used to determine whether the specified data exists in the above-mentioned data set by using a bit array and multiple hash functions.

[0010] As an optional solution, the above-mentioned device further includes: a first determining unit for determining, from the Bloom filters allocated to each of the at least two data sets, a first Bloom filter whose amount of written data is greater than a preset threshold value after the corresponding Bloom filters are allocated to each of the at least two data sets; a second allocating unit for allocating, to the first Bloom filter, an idle Bloom filter whose amount of written data is less than the preset threshold value, after the corresponding Bloom filters are allocated to each of the at least two data sets; a setting unit for allocating, to the first Bloom filter, an idle Bloom filter whose amount of written data is less than the preset threshold value; After each of the at least two data sets is assigned a corresponding Bloom filter, the unexpired stored data in the first data set corresponding to the first Bloom filter is written into the idle Bloom filter, and the data written into the first Bloom filter is set to be written into the idle Bloom filter synchronously; a deleting unit is used to, after each of the at least two data sets is assigned a corresponding Bloom filter, delete the first Bloom filter and set the idle Bloom filter to be the Bloom filter corresponding to the first data set when the unexpired stored data have been written into the idle Bloom filter.

[0011] As an optional solution, the above-mentioned device also includes: a first control unit for, after allocating a target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data, controlling the second Bloom filter allocated to the second data collection to which the above-mentioned second target data belongs when a second data access request triggered for the second target data is obtained, to determine whether the above-mentioned second target data is located in the above-mentioned second data collection; a query unit for, after allocating a target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data, querying the above-mentioned second data collection for the above-mentioned second target data when the above-mentioned first Bloom filter indicates that the above-mentioned second target data is located in the above-mentioned second data collection; a prompt unit for, after allocating a target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data, returning a prompt information of failure to access the above-mentioned second target data to the requester of the above-mentioned second data access request when the above-mentioned second Bloom filter indicates that the above-mentioned second target data is not located in the above-mentioned second data collection.

[0012] As an optional solution, the above-mentioned device also includes: a second determination unit, which is used to determine the second management object from the above-mentioned target number of service management objects after allocating the target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data, when a third data access request triggered by the third target data is obtained; a second control unit, which is used to control the above-mentioned second management object to query the above-mentioned third target data from the third data collection to which the above-mentioned third target data belongs, after allocating the target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data.

[0013] As an optional solution, the above-mentioned device also includes: a second acquisition unit, which is used to obtain the first load value corresponding to the allocated service management objects among the above-mentioned target number of service management objects after allocating the target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data; a third acquisition unit, which is used to reallocate the above-mentioned allocated service management objects and obtain the second load value corresponding to the reallocated service management objects after allocating the target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data and when the above-mentioned first load value indicates that the above-mentioned allocated service management objects are in a load-balanced state; a third allocation unit, which is used to increase or decrease the number of service management objects for allocation and reallocate them until the allocated service management objects are in a load-balanced state after allocating the target number of service management objects to the above-mentioned at least two collection groups to manage the above-mentioned multiple storage data and when the above-mentioned second load value indicates that the above-mentioned reallocated service management objects are still in the above-mentioned load-balanced state.

[0014] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the above-described method for managing stored data.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for managing stored data through the computer program.

[0016] In an embodiment of the present application, a plurality of storage data to be accessed are obtained; the plurality of storage data are hashed and sharded using a first hash function to obtain at least two data sets, wherein the first hash function is used to map each storage data in the plurality of storage data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the storage data in the same data set in the at least two data sets are in the same hash value range; the at least two data sets are hashed and grouped using a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to the data sets in the same set group in the at least two set groups are in the same hash value range; a target number of service management objects are allocated to the at least two set groups to manage the plurality of storage data, wherein the number of set groups in the at least two set groups is determined based on the target number, and the service management objects are management objects responsible for processing access requests for the storage data and providing feedback services.

[0017] The first hash function is used to hash and shard the multiple stored data, distributing the data into at least two data sets. Each data set corresponds to a different first hash value range, ensuring uniform data distribution across the sets. This prevents a single data set from becoming a performance bottleneck due to excessive data.

[0018] Then, the fragmented data set is further hashed and grouped using the second hash function to form at least two groups. Each group corresponds to a different second hash value range, and the data sets contained in the group have similar second hash values. This further achieves balanced data distribution at the group level.

[0019] Further, based on the preset target number (i.e., the number of service management objects), service management objects are dynamically allocated to the collection groups formed after hash grouping. Each service management object is responsible for managing the storage data in one or more collection groups. This dynamic allocation method allows the system to flexibly adjust the number of service management objects according to the current load situation to ensure that each service management object can work efficiently within its processing capacity, thereby achieving the purpose of improving the stability of using service management objects to manage storage data, so as to ensure that when facing a high data access volume, a limited number of service management objects can still provide relatively stable feedback services, thereby achieving the technical effect of improving the management efficiency of storage data, and thus solving the technical problem of low management efficiency of storage data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of an application environment of an optional storage data management method according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a process of an optional method for managing stored data according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional method for managing stored data according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of another optional method for managing stored data according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of another optional method for managing stored data according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of another optional method for managing stored data according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of another optional method for managing stored data according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of an optional storage data management device according to an embodiment of the present application;

[0029] Figure 9 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to one aspect of the embodiment of the present application, a method for managing stored data is provided. Optionally, as an optional implementation, the above-mentioned method for managing stored data can be applied to, but is not limited to, Figure 1 In the environment shown, the environment may include, but is not limited to, a user device 102 and a server 112 . The user device 102 may include, but is not limited to, a display 104 , a processor 106 , and a memory 108 . The server 112 includes a database 114 and a processing engine 116 .

[0033] The specific process can be as follows:

[0034] Step S102: The user device 102 obtains a data access request, wherein the data access request is used to request access to the data to be accessed;

[0035] Step S104 , sending the data access request to the server 112 via the network 110 ;

[0036] In steps S106-S110, the server 112 responds to the data access request, determines the service management object corresponding to the data to be accessed through the processing engine 116, and further controls the service management object to search for the data to be accessed within the corresponding data set, thereby obtaining a query result for the data to be accessed, wherein the service management object corresponds to the set group to which the data to be accessed belongs, and the data set is within the set group to which the data to be accessed belongs;

[0037] In step S112 , the query result is sent to the user device 102 via the network 110 . The user device 102 displays the query result on the display 104 via the processor 106 and stores the query result in the memory 108 .

[0038] remove Figure 1In addition to the examples shown, the above-mentioned terminal device can be a terminal device configured with a target client, which can include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, an MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client can be a video client, an instant messaging client, a browser client, an education client, etc. The above-mentioned network can include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The above-mentioned server can be a single server, or it can be a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made to this in this embodiment.

[0039] Alternatively, as an optional implementation, Figure 2 As shown, the method for managing stored data can be executed by an electronic device, which can be, for example, Figure 1 The user device or server shown in the figure includes the following steps:

[0040] S202, obtaining a plurality of stored data to be accessed;

[0041] S204: Perform hash sharding on the plurality of stored data using a first hash function to obtain at least two data sets, wherein the first hash function is used to map each stored data in the plurality of stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the stored data in the same data set in the at least two data sets are within the same hash value range;

[0042] S206, performing hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to data sets in the same set group in the at least two set groups are within the same hash value range;

[0043] S208, allocate a target number of service management objects to at least two collection groups to manage multiple storage data, wherein the number of collection groups in at least two collection groups is determined based on the target number, and the service management object is a management object responsible for processing access requests for storage data and providing feedback services.

[0044] Optionally, in this embodiment, the above-described method for managing stored data can be applied, but is not limited to, in security scenarios. In security scenarios, for a large number of user access requests, it is often necessary to detect and filter out potentially malicious users in real time, such as blacklisted users. However, since malicious users typically only account for a small fraction of the total user base, using traditional one-by-one detection methods is not only inefficient but also prone to causing service delays or downtime under high traffic conditions.

[0045] In this embodiment, a large number of online user access requests are first obtained. These requests contain user identity information, access content, timestamps, and other data. Using a first hash function, all user access requests are hash-sharded. Since malicious users (such as blacklisted users) account for a relatively small proportion of the total user base, hash sharding can evenly distribute these malicious users across different data sets, avoiding excessive load on a single point.

[0046] Furthermore, the second hash function is used to hash the fragmented data set to further disperse the blacklisted users and other normal users into different sets, ensuring that the number of blacklisted users in each set remains at a low level.

[0047] Based on a preset target number (i.e., the number of service management objects), service management objects are dynamically assigned to the hashed groups. Each service management object is responsible for processing user access requests within one or more groups and executing blacklist detection logic. Because blacklisted users are evenly distributed across the groups, the load on each service management object during blacklist detection is relatively balanced, effectively avoiding the problem of single-point overload.

[0048] The service management object feeds the blacklist detection results back to the system in real time, and the system takes appropriate action based on the feedback (such as blocking access requests from blacklisted users). At the same time, the system can dynamically adjust the number of service management objects based on actual load conditions to ensure efficient resource utilization and stable system operation.

[0049] Optionally, in this embodiment, a series of data items to be processed or accessed are retrieved or received from one or more data sources. These data may be stored in a database, a file system, a memory cache, a remote server, or any other form of data storage system. Among them, to be accessed may refer to data that is about to be read, retrieved, processed, or used in other ways, which may need to be accessed based on a specific request, query, or operation. Multiple stored data may refer to not one, but multiple data items or data sets, which may exist in structured (such as database tables) or unstructured (such as documents, images) forms and are distributed in different storage locations or systems.

[0050] To illustrate further, let's assume an online retail platform stores information on millions of products. When a user searches or browses for a product, the system needs to access multiple stored data points—that is, information about products relevant to the user's query. This data might include product names, descriptions, prices, and inventory levels. These data are stored in multiple database tables and retrieved using complex queries for display to the user.

[0051] Optionally, in this embodiment, by using a first hash function to perform hash sharding on multiple stored data, a large amount of data can be evenly distributed into at least two data sets. Each data set contains data items with similar hash values, which enables independent and parallel operations on each data set in subsequent processing, thereby improving overall processing efficiency. Furthermore, because the data is evenly distributed, the size of each data set is relatively balanced, which helps avoid single-point overload and resource waste.

[0052] To further illustrate, suppose you have a database containing millions of user records, and you want to shard these records by hash value based on the user's ID. You can use a hash function (such as a variant of MD5 or SHA-256) to calculate the hash value of each user ID, and then assign the user records to different data sets based on some part of the hash value (such as the first few digits or characters). For example, all user records with hash values ​​starting with "00" might be assigned to the first data set, while user records with hash values ​​starting with "01" might be assigned to the second data set, and so on.

[0053] In this embodiment, the (first) hash function can be an algorithm or mathematical function that accepts an input (such as the "stored data" in this embodiment) and returns a fixed-size string or number (i.e., a "hash value"), which is theoretically a unique representation of the input data. The characteristic of a hash function is that it can map data of any size to a fixed-size hash value, and that the output hash value is different for different input data (or has an extremely low probability of collision).

[0054] Hash sharding can refer to the process of using a hash function to divide a dataset into multiple smaller, more manageable parts (i.e., "data sets"). Each data set contains data items with similar hash values. A data set can be a collection of data items with similar hash values ​​formed by the hash sharding process.

[0055] Optionally, in this embodiment, by using a second hash function to hash and group at least two data sets, these data sets can be further organized into a smaller number of set groups. Each set group contains data sets with similar second hash values, which enables higher efficiency and better load balancing in subsequent data access, processing, and storage. Furthermore, because related data sets are now organized in the same set group, they can be accessed and processed together more quickly, thereby improving overall performance.

[0056] To further illustrate, let's assume that the user records are divided into 10 data sets using a first hash function. A second hash function can be used to hash these 10 data sets to form fewer sets. For example, the second hash function can be used to calculate a hash value for each data set, and the data sets can be assigned to different sets based on certain parts of these hash values. Ultimately, three sets may be obtained, each containing three to four data sets with similar second hash values.

[0057] The second hash function is similar to the first hash function in that it also maps input data to a fixed-size hash value. However, in this embodiment, its input is a data set that has been sharded by the first hash function, rather than the original stored data.

[0058] Hash grouping can refer to the process of using the second hash function to further divide the data set into multiple "collection groups." Each collection group contains a group of data sets with similar second hash values. A collection group can refer to a higher-level organizational unit formed by the hash grouping process, containing a group of data sets with similar second hash values.

[0059] Optionally, in this embodiment, by allocating a target number of service management objects to at least two collection groups, the data in each collection group can be ensured to be processed and managed promptly and efficiently. Furthermore, because the number of service management objects is determined based on the target number, the number of these objects can be dynamically adjusted based on actual demand and load conditions, achieving better resource utilization and performance optimization. Furthermore, by using service management objects to process access requests for stored data and provide feedback services, overall response speed and user experience can be improved.

[0060] To illustrate further, consider an online shopping system that stores a large amount of product information. This product information is divided into multiple data sets, which are further grouped using hashing to form several collection groups. To manage this product information, a service management object can be assigned to each collection group. When users browse products, their requests are sent to the corresponding service management object, which is responsible for retrieving and providing the required product information. If the system is under heavy load, the number of service management objects can be increased to distribute the load and improve system responsiveness.

[0061] In this embodiment, the target number can be a value preset by the administrator or designer, indicating the number of service management objects that should exist in the system, or it can be determined based on overall performance requirements, load capacity, resource limitations and other factors.

[0062] A service management object is a special software entity or component that handles various operations related to stored data, such as reading, writing, updating, and deleting. It is also responsible for receiving and processing access requests from external systems or users, and providing corresponding feedback services based on the type and content of the request.

[0063] It should be noted that this embodiment significantly improves the efficiency of data storage, access, and management by combining hash sharding, hash grouping, and the use of service management objects. Hash sharding and grouping help evenly distribute data and reduce single-point load, while service management objects are responsible for processing data access requests and providing feedback services. This strategy is particularly suitable for systems that process large amounts of data (such as distributed database systems, cloud storage platforms, big data analysis systems, etc.), and can improve the overall system performance and user experience.

[0064] To further illustrate, the optional Figure 3 As shown, multiple storage data to be accessed (such as storage data 1, storage data 2, storage data 3, etc.) are obtained; the multiple storage data are hashed and sharded by a first hash function to obtain at least two data sets (such as data set A1, data set A2, data set A3, etc.); at least two data sets are hashed and grouped by a second hash function to obtain at least two collection groups (such as collection group B1, collection group B2, etc.); a target number of service management objects are allocated to at least two collection groups (such as allocating service management objects in service management object group C1 to collection group B1, and allocating service management objects in service management object group C2 to collection group B2) to manage multiple storage data, wherein the total number of service management objects in all service management object groups is equal to the target number.

[0065] Through the embodiments provided by the present application, multiple storage data to be accessed are obtained; the multiple storage data are hashed and sharded by a first hash function to obtain at least two data sets, wherein the first hash function is used to map each storage data in the multiple storage data to a corresponding first hash value, different data sets in at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the storage data in the same data set in at least two data sets are in the same hash value range; the at least two data sets are hashed and grouped by a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in at least two data sets to a corresponding second hash value, and the at least two sets Different collection groups in the group correspond to different second hash values, and the second hash values ​​corresponding to the data collections in the same collection group in at least two collection groups are in the same hash value range; a target number of service management objects are allocated to at least two collection groups to manage multiple storage data, wherein the number of collection groups in at least two collection groups is determined based on the target number, and the service management object is the management object responsible for processing access requests for storage data and providing feedback services, thereby achieving the purpose of improving the stability of using service management objects to manage storage data, so as to ensure that when facing a high data access volume, a limited number of service management objects can still provide relatively stable feedback services, thereby achieving the technical effect of improving the management efficiency of storage data.

[0066] As an optional solution, at least two data sets are hashed and grouped using a second hash function to obtain at least two set groups, including:

[0067] S1-1, determining the number of collection groups according to the target number, wherein each of the at least two data sets is configured to allocate at least N service management objects, and the target number is greater than or equal to the product of N and the number of collection groups, where N is an integer greater than 1;

[0068] S1-2, mapping each data set in the at least two data sets to a corresponding second hash value using a second hash function;

[0069] S1-3, allocating each data set in the at least two data sets to a number of set groups according to the hash value range of the corresponding second hash value, to obtain at least two set groups.

[0070] Optionally, in this embodiment, the number of collection groups may be the total number of collection groups calculated based on the target number and the minimum allocation requirement of the service management objects (at least N service management objects are allocated to each data set).

[0071] It should be noted that this embodiment ensures that data is effectively organized and managed at a higher level by determining the number of collection groups based on the target quantity and using a second hash function to map the data collections to hash value ranges. This strategy helps reduce data access latency, improves concurrent processing capabilities during data management, and fully utilizes available service management object resources. Furthermore, by allocating at least N service management objects to each data collection, the reliability and efficiency of data processing requests are guaranteed.

[0072] To further illustrate, let's optionally assume that there are 100 service management objects as the target number, and that each data set needs to be assigned at least 2 service management objects (N=2). If 50 data sets are obtained after processing by the first hash function, then based on these conditions, it can be determined that at least 25 collection groups are required (because 100 service management objects divided by the at least 2 objects required for each collection equals 50, and there are 50 data sets, so at least 25 collection groups are required to accommodate them while ensuring that the data sets in each collection group can obtain the required service management objects). Next, the second hash function is used to map these 50 data sets to hash values, and they are assigned to the 25 collection groups based on these hash values.

[0073] Through the embodiments provided by the present application, the number of collection groups is determined according to the target number, wherein each data set in at least two data sets is set to allocate at least N service management objects, the target number is greater than or equal to the product of N and the number of collection groups, and N is an integer greater than 1; through a second hash function, each data set in at least two data sets is mapped to a corresponding second hash value; according to the hash value range of the corresponding second hash value, each data set in at least two data sets is allocated to the collection groups of the number of collection groups to obtain at least two collection groups, thereby achieving the purpose of improving the concurrent processing capability during data management, thereby realizing the technical effect of improving the management efficiency of stored data.

[0074] As an optional solution, after allocating a target number of service management objects to at least two collection groups to manage the plurality of stored data, the method further includes:

[0075] When a first data access request triggered by the first target data is obtained, a random routing method is adopted to determine the first management object from multiple service management objects assigned to the target set group to which the first target data belongs. The first management object is responsible for processing the first data access request and providing feedback services corresponding to the first target data.

[0076] Optionally, in this embodiment, the first data access request may be initiated by a user or an internal component of the system, and may be a request to read, write or perform other operations on the first target data.

[0077] Optionally, in this embodiment, random routing may be a load balancing strategy that randomly selects a service management object to process a data access request to ensure that the requests are evenly distributed among multiple service management objects.

[0078] It should be noted that this embodiment uses a random routing strategy to select a service management object from the target set to process data access requests. This ensures that requests are evenly distributed across multiple service management objects, thereby avoiding overloading a single object. This strategy helps improve concurrent processing capabilities, response speed, and overall performance of storage data management. Furthermore, because requests are randomly distributed, it also provides a certain degree of fault tolerance and robustness against service management object failures or performance degradation.

[0079] To further illustrate, let's assume there is an online shopping system where product information is stored in multiple data sets, which are grouped into several sets through hashing. Each set is assigned multiple service management objects to process product access requests. When a user attempts to access a specific product (first target data), the system receives a data access request (first data access request). Based on the random routing strategy, the system randomly selects a service management object (first management object) from the set group (target set group) to which the product belongs to process the request. The selected service management object retrieves the product information and returns it to the user as a feedback service.

[0080] Through the embodiments provided in the present application, when a first data access request triggered by the first target data is obtained, a random routing method is adopted to determine the first management object from multiple service management objects assigned to the target set group to which the first target data belongs. The first management object is responsible for processing the first data access request and providing feedback services corresponding to the first target data, thereby achieving the purpose of improving the concurrent processing capability, response speed and overall performance of storage data management, thereby realizing the technical effect of improving the management efficiency of storage data.

[0081] As an optional solution, after performing hash sharding on the plurality of stored data using the first hash function to obtain at least two data sets, the method further includes:

[0082] A corresponding Bloom filter is allocated to each of the at least two data sets, wherein the Bloom filter is used to determine whether specified data exists in the data set by using a bit array and multiple hash functions.

[0083] Alternatively, in this embodiment, a Bloom filter can be understood as a highly space-efficient probabilistic data structure for testing whether an element is a member of a set. It may produce false positive matches but not false negative matches (i.e., if it says an element is not in a set, then the element is definitely not in the set).

[0084] The bit array can be an array of bits (usually 0s and 1s) used in a Bloom filter to store the hash results of the elements in the set. To reduce the false positive rate, the Bloom filter usually uses multiple hash functions to calculate the position of the element in the bit array.

[0085] It should be noted that by assigning a Bloom filter to each data set, this embodiment can quickly and efficiently check whether the specified data is likely to exist in the data set. This helps reduce unnecessary full data search operations, thereby improving the efficiency of data processing and querying.

[0086] To further illustrate, let's assume you have a dataset containing millions of email addresses, and you want to quickly check whether a particular email address is in this dataset. You can assign a Bloom filter to this dataset and use multiple hash functions to map each email address to multiple positions in a bit array. When you want to check whether an email address is in the set, you simply calculate its position in the bit array and check whether those positions are set. If all positions are set, then you can say that the email address is "probably" in the set (due to the possibility of false positives).

[0087] Through the embodiments provided in the present application, a corresponding Bloom filter is assigned to each data set in at least two data sets, wherein the Bloom filter is used to determine whether specified data exists in the data set by using a bit array and multiple hash functions, thereby achieving the purpose of improving the efficiency of data processing and query, thereby realizing the technical effect of improving the management efficiency of stored data.

[0088] As an optional solution, after assigning a corresponding Bloom filter to each of the at least two data sets, the method further includes:

[0089] S2-1, determining a first Bloom filter having a written data amount greater than a preset threshold from Bloom filters allocated to each of the at least two data sets;

[0090] S2-2, allocating an idle Bloom filter to the first Bloom filter, wherein the amount of data written into the idle Bloom filter is less than a preset threshold;

[0091] S2-3, writing the unexpired stored data in the first data set corresponding to the first Bloom filter into the idle Bloom filter, and setting the data to be written into the first Bloom filter to be written into the idle Bloom filter synchronously;

[0092] S2-4: When all the unexpired stored data have been written into the idle Bloom filter, the first Bloom filter is deleted, and the idle Bloom filter is set as the Bloom filter corresponding to the first data set.

[0093] Optionally, in this embodiment, in the Bloom filter, data generally has a certain life cycle or validity. Unexpired stored data refers to data that is still within the validity period and needs to be retained in the Bloom filter.

[0094] It should be noted that in data processing systems, when using Bloom filters to manage large amounts of data, some Bloom filters may become full due to continuous data writing, that is, the amount of data written exceeds a preset threshold. However, since Bloom filters do not support deleting elements from a set, the design of the Bloom filter's bit array and hash function makes deletion operations complex and inaccurate. When attempting to delete an element, it may affect other elements sharing the same bit position, resulting in accidental deletion.

[0095] To overcome the above-mentioned defects, this embodiment first identifies a fully loaded Bloom filter (the first Bloom filter), and then allocates a Bloom filter with free space (the free Bloom filter) to it, and then migrates the data that is still valid in the original Bloom filter (non-expired storage data) to the newly allocated Bloom filter, and ensures that the newly written data is updated to both Bloom filters at the same time. Finally, when the data migration is completed, the original fully loaded Bloom filter is deleted, and the new Bloom filter is set as the Bloom filter of the corresponding data set. In this way, the use of Bloom filters can be effectively managed to avoid performance degradation or increased false positive rate due to the full load of some Bloom filters. By dynamically allocating and migrating data in Bloom filters, the load balance and efficient use of Bloom filters can be maintained, thereby improving the performance and accuracy of data processing.

[0096] To illustrate further, optionally assume there is an email filtering system that uses Bloom filters to quickly check whether an email is spam. Over time, a Bloom filter (the first Bloom filter) becomes full due to the continuous writing of emails. The system identifies this full Bloom filter and finds a Bloom filter with free space (the free Bloom filter). The system then migrates the still valid email addresses (non-expired stored data) in the first Bloom filter to the new Bloom filter, and ensures that newly received email addresses are updated in both Bloom filters at the same time. Once all valid email addresses have been migrated, the system deletes the original first Bloom filter and sets the new Bloom filter as the Bloom filter for the corresponding email collection.

[0097] Through the embodiments provided by the present application, a first Bloom filter having an amount of written data greater than a preset threshold is determined from the Bloom filters allocated to each data set in at least two data sets; an idle Bloom filter is allocated to the first Bloom filter, wherein the amount of written data to the idle Bloom filter is less than the preset threshold; the unexpired storage data in the first data set corresponding to the first Bloom filter is written into the idle Bloom filter, and the data to be written to the first Bloom filter is set to be written into the idle Bloom filter synchronously; when all the unexpired storage data has been written into the idle Bloom filter, the first Bloom filter is deleted, and the idle Bloom filter is set to the Bloom filter corresponding to the first data set, thereby achieving the purpose of improving the performance and accuracy of data processing, thereby realizing the technical effect of improving the management efficiency of stored data.

[0098] As an optional solution, after allocating a target number of service management objects to at least two collection groups to manage the plurality of stored data, the method further includes:

[0099] S3-1, when a second data access request triggered for second target data is obtained, controlling a second Bloom filter allocated to a second data set to which the second target data belongs, to determine whether the second target data is located in the second data set;

[0100] S3-2, when the first Bloom filter indicates that the second target data is located in the second data set, querying the second data set for the second target data;

[0101] S3-3, when the second Bloom filter indicates that the second target data is not located in the second data set, returning prompt information indicating that access to the second target data has failed to the requester of the second data access request.

[0102] Optionally, in this embodiment, a Bloom filter can be used to solve the cache penetration problem. Cache penetration refers to querying a non-existent data item. Since the data item is not in the cache, it is necessary to query the database every time, which puts huge pressure on the database. By using a Bloom filter, you can first quickly determine whether the data item is likely to exist. If it does not exist, it will be returned directly, avoiding invalid queries to the database. However, it should be noted that the Bloom filter has a false positive rate, that is, it may mistakenly believe that a data item exists in the set, so this needs to be taken into account when making decisions using the results of the Bloom filter.

[0103] It should be noted that by using Bloom filters to optimize the processing of data access requests, this embodiment can significantly improve query efficiency and reduce unnecessary database query operations. When the Bloom filter indicates that data may exist in the data set, a further precise query operation is performed to obtain the data; when the Bloom filter indicates that the data is definitely not in the data set, a prompt message indicating access failure can be directly returned without further query operations, thereby saving data processing resources and time costs.

[0104] To further illustrate, it is optional to assume that there is an online shopping system in which product information is stored in multiple data sets, and a Bloom filter is assigned to each data set to optimize query performance. When a user attempts to access a specific product (the second target data), the system receives a data access request (the second data access request). The system first controls the Bloom filter of the data set to which the product belongs (the second Bloom filter) to determine whether the product may exist in the data set. If the Bloom filter indicates that the product may exist, the system will further query the data set to obtain product information; if the Bloom filter indicates that the product is definitely not in the data set, the system will directly return a prompt message indicating that the access failed to the user.

[0105] Through the embodiments provided in the present application, when a second data access request triggered for second target data is obtained, the second Bloom filter assigned to the second data set to which the second target data belongs is controlled to determine whether the second target data is located in the second data set; when the first Bloom filter indicates that the second target data is located in the second data set, the second data set is queried for the second target data; when the second Bloom filter indicates that the second target data is not located in the second data set, a prompt message indicating that access to the second target data has failed is returned to the requester of the second data access request, thereby achieving the purpose of directly returning the prompt message of access failure without performing further query operations, saving data processing resources and time costs, and thus realizing the technical effect of improving the management efficiency of stored data.

[0106] As an optional solution, after allocating a target number of service management objects to at least two collection groups to manage the plurality of stored data, the method further includes:

[0107] S4-1, when a third data access request triggered by third target data is obtained, determining a second management object from a target number of service management objects;

[0108] S4-2: Control the second management object to query the third target data from the third data set to which the third target data belongs.

[0109] It should be noted that this embodiment achieves more flexible and efficient data management by assigning service management objects to different data sets and directing data access requests to the corresponding management objects for processing. This architecture can better cope with high-concurrency access scenarios, improving query response speed and processing capabilities. Furthermore, by optimizing management object selection strategies and utilizing caching mechanisms, the management performance of stored data can be further improved.

[0110] To further illustrate, let's assume a large library system where book information is organized into multiple collections, each with a corresponding service management object. When a user requests information about a specific book (the third target data), the system receives a data access request (the third data access request). The system selects a second management object from the assigned service management objects and then, through that object, initiates a query to the data collection (the third data collection) to which the book belongs, obtaining detailed information about the book.

[0111] Through the embodiments provided by the present application, when a third data access request triggered by third target data is obtained, a second management object is determined from a target number of service management objects; the second management object is controlled to query the third target data from the third data set to which the third target data belongs, thereby achieving the purpose of further improving the management performance of stored data by optimizing the strategy for selecting management objects and utilizing caching mechanisms, thereby realizing the technical effect of improving the management efficiency of stored data.

[0112] As an optional solution, after allocating a target number of service management objects to at least two collection groups to manage the plurality of stored data, the method further includes:

[0113] S5-1, obtaining a first load value corresponding to an allocated service management object among a target number of service management objects;

[0114] S5-2, when the first load value indicates that the allocated service management object is in a load-balanced state, reallocate the allocated service management object and obtain a second load value corresponding to the reallocated service management object;

[0115] S5-3, when the second load value indicates that the reallocated service management objects are still in a load-balanced state, increase or decrease the number of service management objects for allocation, and reallocate until the allocated service management objects are in a load-balanced state.

[0116] Optionally, in this embodiment, the first load value may be used to represent a quantitative indicator of the current workload of the assigned service management object. The second load value may be used to represent a quantitative indicator of the workload of the reallocated service management object. A load balancing state may refer to a state in which the workload is evenly distributed among the service management objects, with no single object being overloaded or idle.

[0117] It should be noted that after allocating a target number of service management objects to at least two collection groups to manage multiple storage data, a series of load balancing steps are also included. For example, this embodiment will obtain the first load values ​​of the currently allocated service management objects, which reflect the workload of each service management object. If the first load values ​​show that these objects are in a load-balanced state, this embodiment will redistribute the service management objects and check the load situation again. If the load is still balanced after redistribution, this embodiment will adjust the number of service management objects for allocation and redistribute them until the load balancing goal is achieved.

[0118] To illustrate further, consider a database management system with 10 database servers serving as management objects, each managing different data sets. The system first allocates these servers and monitors their first load values, indicating an imbalance between some servers with high loads and others with low loads. The system then performs a reallocation, attempting to shift the load from the high-loaded servers to the low-loaded ones. After the reallocation, the system checks the second load value again and finds that, while improved, the imbalance remains. The system then decides to add two additional database servers and redistribute the load, ultimately achieving load balancing.

[0119] Through the embodiment provided by the present application, a first load value corresponding to the allocated service management objects among the target number of service management objects is obtained; when the first load value indicates that the allocated service management objects are in a load-balanced state, the allocated service management objects are reallocated, and a second load value corresponding to the reallocated service management objects is obtained; when the second load value indicates that the reallocated service management objects are still in a load-balanced state, the number of service management objects for allocation is increased or decreased, and reallocated until the allocated service management objects are in a load-balanced state. When the load of a service management object is too high, part of its load can be automatically transferred to other objects, thereby avoiding single point failures and performance bottlenecks. At the same time, when the overall load increases, the load can be shared by increasing the number of service management objects, maintaining the response speed and management stability of the stored data, and thereby achieving the technical effect of efficient management of stored data.

[0120] As an optional solution, and for ease of understanding, the above-described method for managing stored data is applied to a security scenario. In security scenarios, malicious users typically only account for a small fraction of online users; typically, only one out of a hundred users will be blacklisted. This embodiment provides a high-efficiency interception cache system that addresses the limitations of traditional caches in low hit rate scenarios. This system utilizes data segmentation to reduce data pressure on individual machines. Furthermore, the system incorporates distributed Bloom filters to minimize invalid data access.

[0121] It should be noted that this embodiment aims to solve the cache efficiency problem in the case of a large number of request misses. By using a distributed Bloom filter as a pre-filter, the cache system can quickly determine whether a request hits the cache, thereby reducing the number of accesses to the backend key-value storage. Compared with traditional caching solutions, the solution of this embodiment has higher memory utilization and lower space complexity. By generating a distributed Bloom filter for existing data, this embodiment can still maintain efficient operation in the case of large-scale request misses.

[0122] To further illustrate, optionally, the cache system is as follows: Figure 4As shown, when the cache system is deployed on the server side, the full amount of data is sharded into multiple data sets (such as data set A1, data set A2, data set An, etc.) through consistent hashing, and each data set is configured with a corresponding Bloom filter (such as Bloom filter A1, Bloom filter A2, Bloom filter An, etc.); further, the multiple data sets are grouped into multiple set groups through consistent hashing (such as data set A1 and data set An are grouped into set group 2). When the cache system is deployed on the client side, since the client usually has limited resources, the full amount of data is only sharded into multiple data sets (such as data set B1, data set B2, data set Bn, etc.) through consistent hashing, and no corresponding Bloom filters are configured; similarly, the multiple data sets are grouped into multiple set groups through consistent hashing (such as data set B2 is grouped into set group 2).

[0123] Specifically, the core of the cache system lies in data segmentation. This embodiment performs consistent hashing on the full amount of data for segmentation. Generally speaking, 1024 slots (data sets) are sufficient to store all the data. 10 billion data are divided into 1024 slots, and a single slot has less than 10 million keys. When a request comes with a key, the calling end can determine which slot the data belongs to through the same consistent hash. The storage engine (kv) in this embodiment can adopt a storage software such as rocksdb, which can support a table structure through prefixes. This embodiment divides rocksdb into tables, and the number of tables is consistent with the number of slots. In this way, if this embodiment accesses a key value, it can know in which table the data is stored through the slot.

[0124] KV can be understood as the core component of the key-value storage system, responsible for actual data storage and retrieval. Different storage engines can use different data structures and algorithms to provide different performance and functionality. Slot can refer to a basic unit in the storage engine. In this embodiment, each slot corresponds to a specific hash value.

[0125] Optionally, a Bloom filter uses a bit array and multiple hash functions to determine whether an element exists in a set, and has the characteristics of fast query. The above embodiment has divided all the data into slots, and this embodiment generates a corresponding Bloom filter for each slot. There are cases of misjudgment in the Bloom filter, which can be ignored when the misjudgment rate is less than 0.01%. By calculating 10 million key values, 13 hash functions, and 0.01% misjudgment, the memory required to generate the Bloom filter is only 11.7KB. Therefore, each slot only needs to generate a Bloom filter of less than 1M.

[0126] Optionally, after all slots have been allocated in this embodiment, the data is grouped for the remaining 1024 slots. This embodiment again uses consistent hashing to group the data into 64 groups, with each group containing 16 slots. Each group of data corresponds to a group of service machines (service management objects), and each group of service machines maintains the same slots. A group of service machines maintains at least two machines for reliability, allowing for flexible expansion based on the load of grouped data.

[0127] Optionally, in this embodiment, a consistent hashing algorithm is used to map a data item (such as a key) to a point on a hash ring. The hash ring is divided into multiple small intervals or slots. Each node (machine or service instance) is assigned one or more positions on the hash ring and is responsible for processing the data items mapped to its own slot. Another hash function is applied to the data items in each slot to divide them into smaller groups or subsets. This grouping helps to optimize the locality of data, improve query efficiency, and achieve higher levels of load balancing and fault tolerance. When the system receives a request, it determines the group to which it belongs based on the requested data item. From the set of machines responsible for processing the data group, a machine is randomly selected to process the request. The random routing strategy ensures that requests are evenly distributed among the machines in the group to avoid excessive load at a single point.

[0128] Optionally, in this embodiment, the writing process of the cache system is as follows: Figure 5 As shown, the client uses a routing algorithm to calculate the identity of the target server based on specific data attributes (such as a hash value). The client then establishes a direct connection with the target server and makes a data write request. After receiving the write request, the server first writes the data to the storage engine (keyword storage) to ensure data persistence. Next, the server writes relevant data information (such as the key) into the Bloom filter, which is used to quickly determine whether a key exists. To ensure data reliability and fault tolerance, the server propagates the write operation to other machines in the same group. Data replication and synchronization between machines in the group ensure data consistency across multiple replicas. Once the data is successfully written to the local machine and propagated to other machines in the group, the server returns a successful write response to the client. After receiving the successful response, the client confirms that the data has been processed correctly and can proceed with other operations.

[0129] Optionally, in this embodiment, the reading process of the cache system is as follows: Figure 6As shown, the client calculates the identity of the target service machine using a predefined routing algorithm based on the characteristics of the query request (such as hash value or key value). The client directly establishes a connection with the target service machine and sends a query request. The server maintains a Bloom filter to quickly determine whether the queried key exists. When the client sends a query request, the server first uses the Bloom filter to perform a pre-check. If the Bloom filter determines that the key does not exist (miss), it directly returns an empty result to the client, avoiding unnecessary KV queries. If the Bloom filter determines that the key may exist (hit), it continues to query the KV. For keys that the Bloom filter determines may exist, the server will further query the KV storage to obtain the exact data. Once the data is obtained from the KV storage, the server will immediately return the data to the client.

[0130] In a security scenario, the peak number of online single-machine requests can reach 100 billion / minute, but usually less than 40 million hits / minute. The Bloom filter can filter out 99.99% of the miss requests, and the final number of requests for querying the KV is actually only 50 million / minute, greatly reducing the pressure on the KV.

[0131] Optionally, in this embodiment, because online data is constantly being written, data will continue to accumulate. The Bloom filter does not support deletion operations. If data is continuously written to the Bloom filter, it will become full. Therefore, the Bloom filter in this embodiment needs to have elimination logic.

[0132] To further illustrate, the above elimination logic is as follows Figure 7 As shown, a machine is randomly selected from the online host as the candidate host (master), which is responsible for coordinating the update and synchronization of the Bloom filter. All machines in the same group create a new idle Bloom filter to store the latest data information (update) of the slot (data set A1). Ensure the synchronization of the Bloom filters between machines in the same group, including the idle Bloom filter created by the master. The master simultaneously writes online data in its idle Bloom filter (such as writing to Bloom filter A1 and the idle Bloom filter) and pulls the corresponding kv data from the kv sub-table. These kv data are then written to the Bloom filter to ensure that the Bloom filter contains the latest online data. Once the master completes the update of its local Bloom filter, it will synchronize the updated Bloom filter to other machines in the same group.

[0133] Through the embodiments provided by this application, the data query and management efficiency in the distributed storage system is optimized, while the load pressure on key components (such as kv storage) is reduced. By calculating the route to directly select the service machine, the intermediate forwarding links are reduced, and the response speed of the query request is improved. A large number of invalid query requests are pre-filtered using the Bloom filter, thereby reducing the actual query volume of the kv storage and protecting its core resources from excessive consumption. By synchronously updating and merging the Bloom filter, it is ensured that the data in the system is always consistent with the latest online status. By randomly selecting the alternative host (master) and the collaboration mechanism between the machines in the same group, the fault tolerance and scalability of the system are improved.

[0134] By calculating the route, the client can quickly determine the target service machine, avoiding blind forwarding and invalid communication, significantly improving the directionality and processing speed of query requests. The Bloom filter, with its efficient space utilization and query speed, successfully filters out the vast majority of invalid query requests while ensuring a low false alarm rate, significantly reducing the frequency of access to the KV storage. By updating the Bloom filter through synchronous write operations and pulling data from the KV sub-table, the data in the Bloom filter is always synchronized with the online data, providing real-time and accurate query services. The randomly selected master and the collaborative mechanism between machines in the same group enable the system to flexibly respond to node additions and subtractions and load changes, maintaining good scalability and stability.

[0135] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0137] According to another aspect of the embodiment of the present application, a storage data management device for implementing the above storage data management method is also provided. Figure 8 As shown, the device includes:

[0138] A first acquiring unit 802 is configured to acquire a plurality of stored data to be accessed;

[0139] a sharding unit 804 configured to perform hash sharding on the plurality of stored data using a first hash function to obtain at least two data sets, wherein the first hash function is configured to map each stored data in the plurality of stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and first hash values ​​corresponding to stored data in the same data set in the at least two data sets are within the same hash value range;

[0140] a grouping unit 806 configured to perform hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is configured to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to data sets in the same set group in the at least two set groups are within the same hash value range;

[0141] Management unit 808 is used to allocate a target number of service management objects to at least two collection groups to manage multiple storage data, wherein the number of collection groups in at least two collection groups is determined based on the target number, and the service management object is a management object responsible for processing access requests for storage data and providing feedback services.

[0142] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0143] As an optional solution, the grouping unit 806 includes:

[0144] a determining module, configured to determine a number of collection groups based on a target number, wherein each of the at least two data sets is configured to allocate at least N service management objects, and the target number is greater than or equal to the product of N and the number of collection groups, where N is an integer greater than 1;

[0145] a mapping module, configured to map each of the at least two data sets to a corresponding second hash value using a second hash function;

[0146] The allocation module is configured to allocate each data set in the at least two data sets to a set group of the same number according to a hash value range of the corresponding second hash value, so as to obtain at least two set groups.

[0147] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0148] As an optional solution, the device further includes:

[0149] A routing module is used to allocate a target number of service management objects to at least two collection groups to manage multiple stored data. When a first data access request triggered by the first target data is obtained, the module uses a random routing method to determine the first management object from the multiple service management objects allocated to the target collection group to which the first target data belongs. The first management object is responsible for processing the first data access request and providing feedback services corresponding to the first target data.

[0150] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0151] As an optional solution, the device further includes:

[0152] The first allocation unit is used to allocate a corresponding Bloom filter to each of the at least two data sets after performing hash sharding on multiple stored data through a first hash function to obtain at least two data sets, wherein the Bloom filter is used to determine whether specified data exists in the data set by using a bit array and multiple hash functions.

[0153] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0154] As an optional solution, the device further includes:

[0155] a first determining unit configured to, after assigning a corresponding Bloom filter to each of the at least two data sets, determine a first Bloom filter having an amount of written data greater than a preset threshold from the Bloom filters assigned to each of the at least two data sets;

[0156] a second allocating unit, configured to allocate an idle Bloom filter to the first Bloom filter after allocating a corresponding Bloom filter to each of the at least two data sets, wherein an amount of data written to the idle Bloom filter is less than a preset threshold;

[0157] a setting unit, configured to, after assigning a corresponding Bloom filter to each of the at least two data sets, write unexpired stored data in the first data set corresponding to the first Bloom filter into an idle Bloom filter, and to set the data to be written into the first Bloom filter to be synchronously written into the idle Bloom filter;

[0158] A deleting unit is used to, after assigning a corresponding Bloom filter to each data set in at least two data sets, delete the first Bloom filter and set the idle Bloom filter to the Bloom filter corresponding to the first data set when all unexpired stored data have been written into the idle Bloom filter.

[0159] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0160] As an optional solution, the device further includes:

[0161] a first control unit, configured to, after allocating a target number of service management objects to at least two collection groups for managing a plurality of stored data, control a second Bloom filter allocated to a second data collection to which the second target data belongs to determine whether the second target data is located in the second data collection when a second data access request triggered for second target data is obtained;

[0162] a query unit, configured to query the second data set for second target data if the first Bloom filter indicates that the second target data is located in the second data set, after allocating a target number of service management objects to the at least two set groups to manage the plurality of stored data;

[0163] A prompt unit is used to return a prompt message of failure in accessing the second target data to the requester of the second data access request after allocating a target number of service management objects to at least two collection groups to manage multiple storage data, if the second Bloom filter indicates that the second target data is not located in the second data set.

[0164] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0165] As an optional solution, the device further includes:

[0166] a second determining unit configured to, after allocating a target number of service management objects to the at least two collection groups for managing the plurality of stored data, determine a second management object from the target number of service management objects in a case where a third data access request triggered by third target data is obtained;

[0167] The second control unit is configured to control the second management object to query the third data set to which the third target data belongs for the third target data after allocating a target number of service management objects to at least two collection groups to manage multiple storage data.

[0168] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0169] As an optional solution, the device further includes:

[0170] a second acquiring unit, configured to acquire, after allocating a target number of service management objects to at least two collection groups to manage a plurality of stored data, first load values ​​corresponding to allocated service management objects among the target number of service management objects;

[0171] a third acquiring unit, configured to, after allocating a target number of service management objects to the at least two collection groups for managing the plurality of storage data, reallocate the allocated service management objects if the first load value indicates that the allocated service management objects are in a load-balanced state, and acquire a second load value corresponding to the reallocated service management objects;

[0172] The third allocation unit is used to increase or decrease the number of service management objects for allocation and reallocate them until the allocated service management objects are in a load-balanced state after allocating a target number of service management objects to at least two collection groups to manage multiple storage data, when the second load value indicates that the reallocated service management objects are still in a load-balanced state.

[0173] For specific embodiments, reference may be made to the examples shown in the above-mentioned method for managing stored data, which will not be described in detail in this example.

[0174] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned storage data management method is also provided. The electronic device can be, but is not limited to, Figure 1 The user device 102 or server 112 shown in FIG. 1 is used as an example to illustrate the embodiment. Figure 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0175] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0176] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0177] S1, obtaining multiple storage data to be accessed;

[0178] S2, performing hash sharding on the plurality of stored data using a first hash function to obtain at least two data sets, wherein the first hash function is used to map each stored data in the plurality of stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the stored data in the same data set in the at least two data sets are within the same hash value range;

[0179] S3, performing hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to data sets in the same set group in the at least two set groups are within the same hash value range;

[0180] S4, allocates a target number of service management objects to at least two collection groups to manage multiple storage data, wherein the number of collection groups in at least two collection groups is determined based on the target number, and the service management object is a management object responsible for processing access requests for storage data and providing feedback services.

[0181] Alternatively, those skilled in the art will appreciate that Figure 9 The structure shown is for illustration only. Figure 9 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 9 Different configurations shown.

[0182] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the storage data management method and device in the embodiment of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, realizing the above-mentioned storage data management method. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely located relative to the processor 904, and these remote memories can be connected to the electronic device via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks and combinations thereof. Among them, the memory 902 can be used specifically but not limited to store information such as storage data, data sets, collection groups, and service management objects. As an example, such as Figure 9As shown, the memory 902 may include, but is not limited to, the first acquisition unit 802, the slicing unit 804, the grouping unit 806, and the management unit 808 in the data storage management device. In addition, it may also include, but is not limited to, other module units in the data storage management device, which will not be repeated in this example.

[0183] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one embodiment, the transmission device 906 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0184] In addition, the electronic device further includes: a display 908 for displaying information such as the stored data, data sets, set groups, and service management objects; and a connection bus 910 for connecting various module components in the electronic device.

[0185] In other embodiments, the user device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes via network communication. The nodes may form a peer-to-peer network, and any computing device, such as a server, user device, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0186] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instructions containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component and / or installed from a removable medium. When the computer program is executed by a central processing unit, the various functions provided in the embodiments of the present application are performed.

[0187] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0188] It should be noted that the computer system of the electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0189] A computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from the storage unit into random access memory (RAM). The RAM also stores various programs and data required for system operation. The CPU, the read-only memory, and the RAM are connected to each other via a bus. Input / output interfaces (I / O interfaces) are also connected to the bus.

[0190] The following components are connected to the input / output interface: an input section including a keyboard, mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section including a hard disk; and a communication section including a network interface card such as a local area network card and a modem. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. Removable media such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc. are installed in the drive as needed so that computer programs read from them can be installed into the storage section as needed.

[0191] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication portion, and / or installed from a removable medium. When the computer program is executed by a central processing unit, the various functions defined in the system of the present application are performed.

[0192] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations described above.

[0193] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0194] S1, obtaining multiple storage data to be accessed;

[0195] S2, performing hash sharding on the plurality of stored data using a first hash function to obtain at least two data sets, wherein the first hash function is used to map each stored data in the plurality of stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the stored data in the same data set in the at least two data sets are within the same hash value range;

[0196] S3, performing hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to data sets in the same set group in the at least two set groups are within the same hash value range;

[0197] S4, allocates a target number of service management objects to at least two collection groups to manage multiple storage data, wherein the number of collection groups in at least two collection groups is determined based on the target number, and the service management object is a management object responsible for processing access requests for storage data and providing feedback services.

[0198] Alternatively, in the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0199] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the electronic device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0200] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0201] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0202] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed user equipment can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0204] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0206] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for managing stored data, characterized in that: include: Acquire multiple storage data to be accessed; Performing hash sharding on the plurality of stored data using a first hash function to obtain at least two data sets, wherein the first hash function is used to map each stored data in the plurality of stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the stored data in the same data set in the at least two data sets are within the same hash value range; performing hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is used to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to data sets in the same set group in the at least two set groups are within the same hash value range; A target number of service management objects are allocated to the at least two collection groups to manage the multiple storage data, wherein the number of collection groups in the at least two collection groups is determined based on the target number, and the service management objects are management objects responsible for processing access requests for the storage data and providing feedback services.

2. The method according to claim 1, characterized in that The step of performing hash grouping on the at least two data sets by using a second hash function to obtain at least two set groups includes: Determining the number of set groups according to the target number, wherein each of the at least two data sets is configured to allocate at least N of the service management objects, and the target number is greater than or equal to the product of N and the number of set groups, where N is an integer greater than 1; Mapping each of the at least two data sets to the corresponding second hash value using the second hash function; According to the hash value range of the corresponding second hash value, each data set in the at least two data sets is allocated to the number of set groups to obtain at least two set groups.

3. The method according to claim 2, characterized in that After allocating a target number of service management objects to the at least two collection groups to manage the plurality of stored data, the method further includes: When a first data access request triggered by the first target data is obtained, a random routing method is adopted to determine the first management object from multiple service management objects assigned to the target set group to which the first target data belongs. The first management object is responsible for processing the first data access request and providing feedback services corresponding to the first target data.

4. The method according to claim 1, wherein After performing hash sharding on the plurality of stored data using the first hash function to obtain at least two data sets, the method further includes: A corresponding Bloom filter is allocated to each of the at least two data sets, wherein the Bloom filter is used to determine whether specified data exists in the data set by using a bit array and multiple hash functions.

5. The method according to claim 4, characterized in that After assigning a corresponding Bloom filter to each of the at least two data sets, the method further includes: Determining, from the Bloom filters allocated to each of the at least two data sets, a first Bloom filter having an amount of written data greater than a preset threshold; Allocating an idle Bloom filter to the first Bloom filter, wherein the amount of data written to the idle Bloom filter is less than the preset threshold; Writing the unexpired stored data in the first data set corresponding to the first Bloom filter into the idle Bloom filter, and setting the data to be written into the first Bloom filter to be synchronously written into the idle Bloom filter; In a case where all the non-expired stored data have been written into the idle Bloom filter, the first Bloom filter is deleted, and the idle Bloom filter is set as the Bloom filter corresponding to the first data set.

6. The method according to claim 4, characterized in that After allocating a target number of service management objects to the at least two collection groups to manage the plurality of stored data, the method further includes: When a second data access request triggered for second target data is obtained, controlling a second Bloom filter allocated to a second data set to which the second target data belongs, to determine whether the second target data is located in the second data set; In a case where the first Bloom filter indicates that the second target data is located in the second data set, querying the second data set for the second target data; When the second Bloom filter indicates that the second target data is not located in the second data set, prompt information indicating that access to the second target data has failed is returned to the requester of the second data access request.

7. The method according to claim 1, characterized in that After allocating a target number of service management objects to the at least two collection groups to manage the plurality of stored data, the method further includes: determining a second management object from the target number of service management objects when a third data access request triggered by the third target data is obtained; The second management object is controlled to query a third data set to which the third target data belongs for the third target data.

8. The method according to any one of claims 1 to 7, characterized in that After allocating a target number of service management objects to the at least two collection groups to manage the plurality of stored data, the method further includes: Obtaining first load values ​​corresponding to allocated service management objects among the target number of service management objects; When the first load value indicates that the allocated service management object is in a load balancing state, reallocating the allocated service management object and obtaining a second load value corresponding to the reallocated service management object; When the second load value indicates that the reallocated service management objects are still in the load balancing state, the number of service management objects for allocation is increased or decreased, and reallocation is performed until the allocated service management objects are in the load balancing state.

9. A storage data management device, characterized in that: include: A first acquiring unit, configured to acquire a plurality of stored data to be accessed; a sharding unit, configured to perform hash sharding on the plurality of stored data using a first hash function to obtain at least two data sets, wherein the first hash function is configured to map each stored data in the plurality of stored data to a corresponding first hash value, different data sets in the at least two data sets correspond to different first hash values, and the first hash values ​​corresponding to the stored data in the same data set in the at least two data sets are within the same hash value range; a grouping unit, configured to perform hash grouping on the at least two data sets using a second hash function to obtain at least two set groups, wherein the second hash function is configured to map each data set in the at least two data sets to a corresponding second hash value, different set groups in the at least two set groups correspond to different second hash values, and the second hash values ​​corresponding to data sets in the same set group in the at least two set groups are within the same hash value range; A management unit is used to allocate a target number of service management objects to the at least two collection groups to manage the multiple storage data, wherein the number of collection groups in the at least two collection groups is determined based on the target number, and the service management objects are management objects responsible for processing access requests for the storage data and providing feedback services.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program is executed by an electronic device to perform the method according to any one of claims 1 to 8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images