A Membership Determination Structure Budget-Safe Switching Method in Search Aggregation Requests

By dynamically switching between exact and approximate sets in the search engine using an adaptive set structure, combined with a cuckoo filter and a request-level memory circuit breaker, the accuracy issues of set structures in small cardinality scenarios and the memory controllability issues in large cardinality scenarios are resolved, thereby improving the system's memory management and stability.

CN123068640APending Publication Date: 2026-11-10JIANGSU DAMENG DATABASE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611145592.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-30
Publication Date
2026-11-10

Smart Images

  • Figure CN123068640A_ABST
    Figure CN123068640A_ABST
Patent Text Reader

Abstract

This invention discloses a budget-safe switching method for member determination structure in search aggregation requests, belonging to the field of search engine optimization technology. The method creates an adaptive set structure associated with the current search aggregation request and links it to a request-level memory circuit breaker. In the small cardinality stage, an exact set is used for member determination. When the switching condition is met, the budget memory usage is verified by the request-level memory circuit breaker, and the exact set is switched to an approximate set. In the reduce phase of distributed aggregation, high-frequency words are actively identified and dynamically pruned based on the approximate set. This invention balances the accuracy of small cardinality determination with the controllability of memory in large cardinality aggregation scenarios, improving the search performance of search engine services.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the fields of search engines, distributed data analysis, memory resource control and approximate set determination technology, specifically relating to a method for safe switching of member determination structure budget in search aggregation requests, electronic devices and computer-readable storage media. Background Technology

[0002] In search engine aggregation analytics, rare word aggregation is used to discover low-frequency items in a document collection. For example, in scenarios such as log analysis, anomaly tag discovery, and low-frequency error code statistics, the system needs to maintain a candidate set of non-rare items to store and determine which items have been identified as non-rare. For each encountered item, the system needs to quickly determine whether it already exists in this set to decide whether to increment the count or mark it as a rare item.

[0003] In a distributed search architecture, multiple shards generate local aggregation results. A coordinating node merges these results during the reduce phase (the process of aggregating and merging the local aggregation results calculated from multiple data shards in a distributed aggregation scenario). For a candidate term (a data item used for statistics and grouping during the aggregation process), if its number of documents after merging across multiple shards exceeds the maximum document count threshold, that term should no longer be output as a low-frequency item. To avoid repeatedly processing these "determined non-low-frequency" terms, the system typically needs to maintain a membership decision set.

[0004] Common practices in existing technologies include:

[0005] Use an exact set, such as HashSet. This method is accurate and has no false positives, but when the cardinality of terms is large, memory usage increases linearly with the number of elements, which can easily lead to excessive memory consumption in a single aggregation request.

[0006] Using a traditional Bloom filter is a good option. Bloom filters are space-efficient, but they typically require capacity estimation; the false positive rate worsens when the actual number of insertions exceeds the estimated capacity. Using a scalable Bloom filter requires maintaining multiple layers of byte arrays, which may need to be accessed during queries, and there is usually a lack of fine-grained coordination between capacity expansion and request-level memory budget.

[0007] Using a fixed approximate set structure. This approach incurs the fixed cost of the approximate structure in small cardinality scenarios and introduces unnecessary misjudgments; in large cardinality scenarios, if the capacity is insufficient, it may lead to an increased misjudgment rate or insertion failures that are difficult for the memory management mechanism to detect.

[0008] The above solution has the following problems:

[0009] 1. The fixed set structure makes it impossible to balance the accuracy with small cardinality and the memory controllability with large cardinality.

[0010] 2. The growth of approximate structure capacity is out of sync with the memory budget of aggregated requests, which can easily cause request-level memory overflow in complex aggregated requests.

[0011] 3. Traditional Bloom filters are not flexible enough in scenarios with unknown cardinality, dynamic expansion, and merging of split reducers.

[0012] 4. The aggregation reduce phase lacks an efficient filtering collection structure that can switch and expand online based on the actual term base and register memory changes in real time.

[0013] An effective solution is needed to address these issues. Summary of the Invention

[0014] Purpose of the invention: This invention provides a method for safe switching of member determination structure budget in search aggregation requests, which solves the problems that the member determination set in search aggregation requests should not introduce misjudgments in small cardinality scenarios, that memory cannot grow indefinitely in large cardinality scenarios, and that the expansion process of approximate sets is difficult to be perceived by the request-level memory budget.

[0015] Technical solution: A method for secure switching of membership determination structure budget in search aggregation requests, comprising the following steps:

[0016] Create an adaptive set structure and initialize the running parameters. The adaptive set structure is initially in exact mode and maintains an exact set in exact mode.

[0017] Receive the aggregate items to be processed, generate member identifiers based on the aggregate items to be processed, write the member identifiers into the precise set in precise mode, and trigger the set structure transformation process when the cardinality of the members in the precise set reaches the specified cardinality threshold.

[0018] In the set structure transformation process, the estimated memory change is submitted to the request-level memory circuit breaker corresponding to the current search aggregation request through the request-level memory callback function. After the verification is passed, an approximate set is created. The approximate set is a probabilistic data structure used to perform the existence determination of member identifiers with a configurable false positive rate. The member identifiers in the exact set are inserted into the approximate set, and the adaptive set structure is switched to the approximate mode.

[0019] In approximation mode, the existence of member identifiers of newly added aggregate items is determined by approximation set. If the member identifier does not exist, it is written into the approximation set. If the writing fails, a new approximation set is created to complete the writing.

[0020] In the reduce phase of distributed aggregation, high-frequency words are actively identified and dynamically pruned based on the approximate set.

[0021] Furthermore, the operating parameters of the adaptive aggregation structure include a cardinality threshold, a false positive rate parameter, a request-level memory callback function, and a budget threshold. The cardinality threshold is used to determine whether the adaptive aggregation structure needs to switch from precise mode to approximate mode. The false positive rate parameter is used to determine the configuration parameters of the approximate aggregation structure, thereby controlling the false positive probability of the approximate aggregation structure. The request-level memory callback function is used to synchronously feed back the expected memory usage to the request-level memory circuit breaker for real-time memory protection. The budget threshold is used to limit the maximum memory allowed for the current aggregation request. When the expected memory usage exceeds the budget threshold, the aggregation structure switch is triggered in advance or the corresponding memory protection strategy is executed.

[0022] Furthermore, the member identifier is obtained by performing a hash operation, encoding conversion, or fingerprint extraction on the aggregate item to be processed.

[0023] Furthermore, the exact set is a hash set; the operating parameters of the adaptive set structure also include a hash seed, which is used to ensure the consistency of the member identifier hash value and the approximate set structure fingerprint generation process, so that the same member can generate a stable and consistent hash result throughout the entire request lifecycle.

[0024] Furthermore, the approximate set is a cuckoo filter.

[0025] Furthermore, the existence of member identifiers of newly added aggregate items is determined using an approximate set. If an identifier is not found, it is written to the approximate set. If the writing fails, a new approximate set is created to complete the writing process. This includes:

[0026] Generate a fingerprint based on the member identifier corresponding to the newly added aggregation item;

[0027] The two candidate buckets corresponding to the fingerprint are determined according to the hash rule;

[0028] If a fingerprint exists in either candidate bucket, the member identifier is determined to already exist and no writing is performed; if no fingerprint exists in either candidate bucket, the fingerprint is written to the current cuckoo filter.

[0029] If writing to the current cuckoo filter fails, create a new cuckoo filter and add it to the filter list, and write the fingerprint to the new cuckoo filter.

[0030] Furthermore, in approximation mode, multiple cuckoo filters in the filter list jointly participate in the membership determination of newly added aggregate items.

[0031] Furthermore, in the reduce phase of distributed aggregation, high-frequency words are actively identified and dynamically pruned based on the approximate set, including:

[0032] Merge the adaptive set structures returned by each shard, and merge and count the aggregate buckets with the same member identifier; if the merged count is not greater than the low-frequency threshold and the corresponding member identifier does not exist in the cuckoo filter, then the member identifier is used as a low-frequency candidate output; if the merged count is greater than the low-frequency threshold, then the member identifier is added to the cuckoo filter to prune subsequent identical member identifiers.

[0033] Furthermore, the lifecycle of the adaptive set structure is consistent with the lifecycle of the current search aggregation request; after the current search aggregation request ends, the adaptive set structure is released and recreated in subsequent search aggregation requests.

[0034] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the above-described method for safe switching of member determination structure budget in search aggregation requests.

[0035] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for safe switching of member determination structure budget in search aggregation requests.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for safe switching of member determination structure budget in search aggregation requests.

[0037] Beneficial effects:

[0038] (1) This invention creates a request-level adaptive set structure during the execution of the search aggregation request. It maintains the precise set for member determination during the small cardinality stage, which can avoid misjudgment and ensure the accuracy of member determination. When the cardinality of members reaches the cardinality threshold, the member identifiers in the precise set are migrated to the approximate set, and the adaptive set structure is switched to the approximate mode. This avoids the precise set from continuously expanding as the number of aggregation items increases, and improves the memory controllability in large cardinality aggregation scenarios.

[0039] (2) By associating the mode conversion of the adaptive set structure with the request-level memory circuit breaker, the present invention incorporates the memory overhead of the member decision structure into the global memory budget control system of the search aggregation request. When the memory usage reaches the dangerous threshold, the request-level protection is triggered in time, which effectively avoids the entire node from crashing due to the memory over-limit of individual aggregation requests, and improves the memory security and operational stability of the system.

[0040] (3) In the distributed aggregation reduce stage, the present invention merges the adaptive set structure returned by multiple data shards, and combines the merge count and member judgment results to filter low-frequency candidates. Aggregate items exceeding the low-frequency threshold are recorded so that subsequent identical aggregate items can be pruned, thereby reducing the overhead of repeated processing and improving the execution stability of distributed scarce word aggregation. Attached Figure Description

[0041] Figure 1 Flowchart of the budget-safe switching method for adaptive collection structure in search aggregation requests.

[0042] Figure 2 This is a schematic diagram of the request-level memory linkage process in this invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to illustrate the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0044] This invention provides a method and system for secure switching of member determination structure budget in search aggregation requests, applied to the process of a search engine executing search aggregation requests. In this invention, an aggregation item refers to a data item used for statistics and grouping during the search aggregation process, abbreviated as "term". An aggregation item can specifically manifest as field values, tags, log keywords, error codes, or other data content participating in aggregation statistics.

[0045] The system in this invention refers to a search system or search engine server used to execute search aggregation requests. It includes an aggregation execution instance, an adaptive set structure, a set structure management module, and a request-level memory circuit breaker. The system uses an exact set structure during the small cardinality phase, and automatically converts to an approximate set structure after the cardinality exceeds a threshold. The approximate set structure refers to a data structure that uses probability to determine member existence, and it submits the memory changes caused by conversion, expansion, and merging operations to the request-level memory circuit breaker. The adaptive set structure in this invention refers to a request-level runtime set structure created during the execution of the current search aggregation request. In the embodiments, a hash set is maintained to store member information in exact mode, and a cuckoo filter is used to store member information in approximate mode. The lifecycle of the adaptive set structure is consistent with the lifecycle of the current search aggregation request. After the current search aggregation request ends, the adaptive set structure is released, and subsequent search aggregation requests create new adaptive set structures.

[0046] like Figure 1 As shown, the method of the present invention includes the following steps S101 to S107. Figure 2This is used to illustrate the request-level memory linkage process involved in the execution of this method. This request-level memory linkage process runs through the creation of the adaptive set structure, the transformation of the set structure, the expansion of the cuckoo filter, and the distributed aggregation and merging process.

[0047] S101, Create an adaptive set structure for member determination, and set the running parameters of the adaptive set structure.

[0048] When a search engine receives a search aggregation request, it creates a corresponding aggregation execution instance and an adaptive set structure for member determination for that aggregation execution instance.

[0049] In one implementation, the operating parameters of the adaptive aggregate structure may include a cardinality threshold, a false positive rate parameter, a hash seed, a request-level memory callback function, and a budget threshold. The cardinality threshold is used to determine whether the adaptive aggregate structure needs to switch from exact mode to approximate mode. The false positive rate parameter is used to determine configuration parameters such as capacity planning, fingerprint bit width, and bucket capacity of the approximate aggregate structure, thereby controlling the false positive probability of the approximate aggregate structure. The hash seed ensures the consistency of member identifier hash values ​​and the fingerprint generation process of the approximate aggregate structure, enabling the same member to generate stable and consistent hash results throughout the entire request lifecycle. The request-level memory callback function synchronously feeds back the estimated memory change to the request-level memory circuit breaker when the aggregate structure requests memory, so as to know the memory usage of the current request in real time and perform budget control. The budget threshold limits the maximum memory allowed for the current aggregation request; when the estimated memory usage exceeds the budget threshold, the aggregate structure switch can be triggered in advance or an appropriate memory protection strategy can be executed.

[0050] For example, in one implementation, the operating parameters can be set as follows: the cardinality threshold is 10,000, indicating that the structure switch is started when the number of members in the exact set reaches 10,000; the false positive rate parameter is 0.1%, used to control the false positive probability of the approximate set structure; the hash seed uses a 64-bit random initialization value to generate stable and consistent member hash values ​​and fingerprints; the request-level memory callback function is used to update the memory usage in the request-level memory circuit breaker in real time when the set structure is expanded or switched; the budget threshold is set to 64MB to limit the maximum memory that the current aggregation request can use. When the expected memory usage exceeds the budget threshold, the set structure switch can be triggered in advance or the corresponding memory protection strategy can be executed.

[0051] After initialization, the adaptive set structure is in exact mode and maintains a hash set in exact mode. The hash set is used to store the member identifiers corresponding to the aggregate items. Because the hash set is an exact set structure, it can provide false-error-free member determination results during the small cardinality stage.

[0052] like Figure 2As shown, when creating the aggregation execution instance corresponding to the search aggregation request, the association between the adaptive set structure and the request-level memory circuit breaker is established at the same time. The request-level memory callback function submits the memory changes generated by the adaptive set structure during creation, set structure transformation, cuckoo filter expansion, and result merging to the request-level memory circuit breaker corresponding to the current search aggregation request.

[0053] S102, receive the aggregate item to be processed, generate a member identifier based on the aggregate item to be processed, and write the member identifier into the hash set in exact mode.

[0054] During the execution of a search aggregation request, the system continuously receives aggregate items to be processed and generates member identifiers based on these items. The member identifiers, calculated from the aggregate items, are used in set storage, member determination, and set structure transformation. Member identifiers can be obtained from aggregate items through hash operations, encoding conversion, or fingerprint extraction.

[0055] In one implementation, when the aggregate item to be processed is a string, byte sequence, or numeric type, the system calculates the corresponding hash value using a preset hash algorithm and uses this hash value as a member identifier. In another implementation, the system may also perform encoding conversion, fingerprint extraction, or other mapping processing on the aggregate item to be processed to obtain the member identifier. Subsequent collection storage, member determination, and collection structure conversion are all processed based on the member identifier.

[0056] When the adaptive set structure is in exact mode, the system writes the member identifier to the hash set. If the member identifier already exists in the hash set, it is determined that the corresponding aggregate item has been recorded; if the member identifier does not exist in the hash set, the member identifier is added to the hash set.

[0057] In rare word aggregation scenarios, the system can continuously count the number of documents for each aggregation item during the aggregation calculation process. For data items that meet preset conditions, the system records them in an adaptive set structure for subsequent aggregation processing.

[0058] S103: Count the cardinality of members in the hash set. When the cardinality of members reaches the cardinality threshold, the set structure transformation process is triggered.

[0059] The system continuously monitors the cardinality of members in the hash set. Cardinality is the number of distinct member identifiers currently written to the hash set. When the cardinality is below the cardinality threshold, the adaptive set structure continues to operate in exact mode and uses the hash set for member determination and writing. When the cardinality reaches or exceeds the cardinality threshold, the system triggers a set structure transformation process, converting the adaptive set structure from exact mode to approximate mode. This judgment process ensures accurate member determination using the hash set during the low cardinality phase and prevents the hash set from continuously consuming more memory as the number of aggregated items increases during the high cardinality phase.

[0060] S104, in the set structure transformation process, create a cuckoo filter, insert the member identifiers in the hash set into the cuckoo filter, and switch the adaptive set structure to the approximate mode.

[0061] The estimated memory change is submitted to the request-level memory circuit breaker corresponding to the current search aggregation request via a request-level memory callback function. After successful verification, a cuckoo filter is created, and the member identifiers from the hash set are inserted into the cuckoo filter. The estimated memory change can be estimated based on a cardinality threshold and a false positive rate parameter, using the capacity calculation formula of the cuckoo filter to derive the theoretical memory upper limit. Alternatively, it can be determined using other technical means, which are not limited in this invention. If the estimated memory usage exceeds the budget threshold, potentially leading to insufficient available system memory, request-level memory protection is triggered; if the estimated memory usage does not exceed the verification condition, the request is allowed to continue execution. Request-level memory protection may include refusing to allocate more memory, interrupting the current aggregation operation, terminating the current search aggregation request, or returning memory exception information.

[0062] If the verification passes, a cuckoo filter is created and the structure is switched. Specifically, the system iterates through the member identifiers in the hash set and generates fingerprints suitable for storage in the cuckoo filter based on the member identifiers. The cuckoo filter stores member information based on buckets and fingerprints, with each fingerprint corresponding to two candidate buckets according to the hash rules. When inserting a fingerprint, it is inserted into either of the two candidate buckets.

[0063] After the member identifiers in the hash set are migrated, the hash set is cleared, and the adaptive set structure is switched to approximate mode. After the switch, the adaptive set structure no longer relies on the hash set to perform member determination, but instead performs member determination and writing through the cuckoo filter.

[0064] S105, in approximation mode, performs membership determination and writing of new aggregate items through the cuckoo filter, and expands the cuckoo filter when the cuckoo filter capacity is insufficient.

[0065] After the adaptive set structure enters approximation mode, the system continues to receive new aggregate items, generates corresponding member identifiers based on the new aggregate items, and then generates fingerprints based on the member identifiers. In approximation mode, the adaptive set structure includes a filter list, which contains one or more cuckoo filters. The fingerprint corresponding to a new aggregate item is first written to the last cuckoo filter in the filter list.

[0066] The member determination and writing of new aggregation items are as follows: Generate a fingerprint based on the member identifier corresponding to the new aggregation item; determine the two candidate buckets corresponding to the fingerprint according to the hash rule; if the fingerprint exists in either candidate bucket, it is determined that the member identifier already exists and no writing is performed; if the fingerprint does not exist in either candidate bucket, the fingerprint is written to the current cuckoo filter; if writing to the current cuckoo filter fails, a new cuckoo filter is created and added to the filter list, and the fingerprint is written to the new cuckoo filter.

[0067] When the current cuckoo filter insertion fails and a new cuckoo filter needs to be created, the system estimates the memory required for the new cuckoo filter and reads the available system memory before creating the new filter. A preliminary verification is also performed. If the verification passes, the new cuckoo filter is created; otherwise, request-level memory protection is triggered. After the new cuckoo filter is created, the system submits the memory size of the new cuckoo filter to the request-level memory circuit breaker through a request-level memory callback function.

[0068] In approximation mode, multiple cuckoo filters in the filter list participate in the membership determination of newly added aggregate items. By adding new cuckoo filters, the entire set structure can be rebuilt as the cardinality continues to grow, while the increased memory generated by the expansion can be detected by the request-level memory circuit breaker.

[0069] S106 In a distributed search aggregation scenario, multiple data shards can perform local aggregation calculations separately and return the aggregation results of scarce words and an adaptive set structure.

[0070] In the distributed aggregation reduce phase, the merge execution node receives the scarce word aggregation results and adaptive set structures returned by multiple data shards, and merges the adaptive set structures returned by multiple data shards. When the adaptive set structures returned by multiple data shards are in an approximate pattern, the merge execution node can merge the cuckoo filters returned by multiple data shards according to the bucket and fingerprint granularity in the cuckoo filter, and perform membership determination based on the merged cuckoo filter. Since the cuckoo filter internally stores buckets and fingerprints, rather than the original aggregation items, merging can be performed based on buckets and fingerprints in the reduce phase without transmitting or restoring the original aggregation items. The reduce phase is essentially merging the fingerprint sets stored in each data shard to obtain the merged filter state.

[0071] After the adaptive set structure merging is completed, the merge execution node counts the identical aggregate items returned by multiple data shards to obtain the number of merged documents. When the number of merged documents does not exceed the low-frequency threshold and the aggregate item does not match the merged adaptive set structure, the aggregate item is output as a low-frequency candidate. When the number of merged documents exceeds the low-frequency threshold, the aggregate item is added to the merged adaptive set structure, causing subsequent identical aggregate items to be pruned.

[0072] During the distributed aggregation reduce phase, if the merging of the adaptive set structure causes changes in memory usage, the system can also submit these changes to the request-level memory circuit breaker through a request-level memory callback function, so that the result merging process is also included in the memory budget management of the current search aggregation request.

[0073] like Figure 2 As shown, if the aggregation task completes normally, the system outputs the aggregation result; if request-level memory protection is triggered, the system returns the corresponding exception information. This protection mechanism can prevent a single search aggregation request from consuming too many system resources and reduce the risk of node memory overflow.

[0074] The above steps can be implemented by a search engine server, a coordinating node or data node in a distributed search system, or a server cluster that executes search aggregation tasks. The method can also be implemented by a computer program stored in a computer-readable storage medium and executed by a processor.

[0075] This invention employs a cuckoo filter as the basic structure for approximating patterns, primarily for the following reasons: Traditional Bloom filters typically rely on estimated capacity, and the false positive rate continuously deteriorates once the actual number of insertions exceeds the design capacity. Furthermore, they require filters participating in the merging process to have the same length, hash function, and parameters, and the merging is completed via a bitwise OR operation. Even after merging, the false positive rate is difficult to control precisely when the actual cardinality exceeds expectations. In contrast, the cuckoo filter can sense capacity pressure through insertion failure signals and trigger expansion, thus exhibiting better adaptability in scenarios involving data growth and distributed aggregation and merging.

[0076] The Cuckoo Filter is based on fingerprint and candidate bucket storage, and insertion failures are explicitly detectable. When an insertion fails, the system can immediately add a new filter and simultaneously register the new memory. The Cuckoo Filter does not directly store the original element; instead, it stores a short fingerprint obtained by hashing the element—a portion of the element's hash value. Each fingerprint corresponds to two candidate buckets according to the hash rules. Insertion, query, and deletion operations all revolve around the fingerprint; therefore, the actual storage unit inside the filter is "bucket + fingerprint," not the original element. In the reduce phase of aggregation, the Cuckoo Filters returned by each shard can be merged at the "bucket + fingerprint" granularity without transferring or restoring the original elements. Since the fingerprint itself is the unique storage object in the filter, the reduce phase essentially merges the fingerprint sets stored in each shard to obtain the global filter state. This characteristic makes the Cuckoo Filter naturally suitable as a deduplication data structure in distributed aggregation scenarios.

[0077] Therefore, this invention does not simply use the cuckoo filter, but rather treats it as an "expandable approximate set form after exceeding the threshold" and links it with the request-level memory circuit breaker to form a dynamic capacity management mechanism.

[0078] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program that, when executed by the processor, implements the aforementioned method for secure switching of the membership determination structure budget in a search aggregation request. The electronic device can be a search engine server, a coordinating node or data node in a distributed search system, or a server cluster executing search aggregation tasks. The computer program may include an aggregation executor, a membership determination structure management program, and a request-level memory circuit breaker program.

[0079] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for safe switching of member determination structure budget in search aggregation requests.

[0080] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for safe switching of member determination structure budget in search aggregation requests.

Claims

1. A method for secure switching of member determination structure budget in search aggregation requests, characterized in that, Includes the following steps: Create an adaptive set structure and initialize the running parameters. The adaptive set structure is initially in exact mode and maintains an exact set in exact mode. Receive the aggregate items to be processed, generate member identifiers based on the aggregate items to be processed, write the member identifiers into the precise set in precise mode, and trigger the set structure transformation process when the cardinality of the members in the precise set reaches the specified cardinality threshold. In the set structure transformation process, the estimated memory change is submitted to the request-level memory circuit breaker corresponding to the current search aggregation request through the request-level memory callback function. After the verification is passed, an approximate set is created. The approximate set is a probabilistic data structure used to perform the existence determination of member identifiers with a configurable false positive rate. The member identifiers in the exact set are inserted into the approximate set, and the adaptive set structure is switched to the approximate mode. In approximation mode, the existence of member identifiers of newly added aggregate items is determined by approximation set. If the member identifier does not exist, it is written into the approximation set. If the writing fails, a new approximation set is created to complete the writing. In the reduce phase of distributed aggregation, high-frequency words are actively identified and dynamically pruned based on the approximate set.

2. The method according to claim 1, characterized in that, The operating parameters of the adaptive aggregation structure include cardinality threshold, false positive rate parameter, request-level memory callback function, and budget threshold. Among them, the cardinality threshold is used to determine whether the adaptive aggregation structure needs to switch from precise mode to approximate mode; the false positive rate parameter is used to determine the configuration parameters of the approximate aggregation structure, thereby controlling the false positive probability of the approximate aggregation structure; the request-level memory callback function is used to synchronously feed back the expected memory usage to the request-level memory circuit breaker for real-time memory protection; the budget threshold is used to limit the maximum memory allowed for the current aggregation request. When the expected memory usage exceeds the budget threshold, the aggregation structure switch is triggered in advance or the corresponding memory protection strategy is executed.

3. The method according to claim 1, characterized in that, The member identifier is obtained by performing hash operations, encoding conversion, or fingerprint extraction on the aggregate item to be processed.

4. The method according to claim 1, characterized in that, The exact set is a hash set; the operating parameters of the adaptive set structure also include a hash seed, which is used to ensure the consistency of the member identifier hash value and the fingerprint generation process of the approximate set structure, so that the same member can generate a stable and consistent hash result throughout the entire request lifecycle.

5. The method according to claim 1, characterized in that, The approximate set is the cuckoo filter; The existence of member identifiers of newly added aggregate items is checked using an approximate set. If an identifier is not found, it is written to the approximate set. If the writing fails, a new approximate set is created to complete the writing process. This includes: Generate a fingerprint based on the member identifier corresponding to the newly added aggregation item; The two candidate buckets corresponding to the fingerprint are determined according to the hash rule; If a fingerprint exists in either candidate bucket, the member identifier is determined to already exist and no writing is performed; if no fingerprint exists in either candidate bucket, the fingerprint is written to the current cuckoo filter. If writing to the current cuckoo filter fails, create a new cuckoo filter and add it to the filter list, and write the fingerprint to the new cuckoo filter.

6. The method according to claim 5, characterized in that, In approximation mode, multiple cuckoo filters in the filter list participate in the membership determination of newly added aggregate items.

7. The method according to claim 1, characterized in that, In the reduce phase of distributed aggregation, high-frequency words are actively identified and dynamically pruned based on the approximate set, including: Merge the adaptive set structures returned by each shard, and merge and count the aggregate buckets with the same member identifier; if the merged count is not greater than the low-frequency threshold and the corresponding member identifier does not exist in the cuckoo filter, then the member identifier is used as a low-frequency candidate output; if the merged count is greater than the low-frequency threshold, then the member identifier is added to the cuckoo filter to prune subsequent identical member identifiers.

8. The method according to claim 1, characterized in that, The lifecycle of the adaptive collection structure is the same as the lifecycle of the current search aggregation request; after the current search aggregation request ends, the adaptive collection structure is released and recreated in subsequent search aggregation requests.

9. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the programs are stored in memory and configured to be executed by one or more processors, wherein when the programs are executed by the processors, they implement the member determination structure budget safe switching method in the search aggregation request of any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the member determination structure budget safe switching method in the search aggregation request of any one of claims 1-8.