A collaborative control method based on a dynamically evolving hybrid learning index structure

Abstract

The application belongs to the technical field of database index, and specifically discloses a collaborative control method based on a dynamically evolving hybrid learning index structure. Through the application, a multi-granularity semantic fusion mapping module is used to convert an input variable-length string set into an integer value of a target bit, and a target hybrid index is constructed in batches according to the integer value; a root learning index node prediction slot is used to traverse the target hybrid index in a recursive descending manner, and a data query task is executed at the target node position. Through the above method, the multi-granularity semantic fusion mapping module is introduced to convert the ladder-shaped distribution of variable-length strings into uniform linear integer values, strictly retain the lexicographical order of the strings, construct the target hybrid index, predict the slot, locate the target node position, and query the data, so that the efficiency of querying the data can be effectively improved, and the core requirements of high-performance in-memory databases and key-value storage systems can be met.

Description

Technical Field

[0001] This application belongs to the field of database indexing technology, and more specifically, relates to a collaborative control method based on a dynamically evolved hybrid learning index structure. Background Technology

[0002] With the rapid development of emerging technologies such as big data, cloud computing, and artificial intelligence, the scale of data is growing explosively. Variable-length string keys have become the mainstream data key type, widely used in scenarios such as URLs, user unique identifiers, file paths, log TraceIDs, biological sequences, and object storage paths. These variable-length strings typically have characteristics such as long common prefixes, long tail distributions, and variable lengths, posing a serious challenge to traditional indexes and existing learning indexes.

[0003] Traditional indexes (such as B-trees, Adaptive Radix Trees (ART), and Height-Optimized Prefix Trees (HOT)) have significant drawbacks in variable-length string scenarios: First, string comparison operations are costly, and frequent pointer jumps lead to poor cache locality and high access latency; second, the presence of long common prefixes increases the height of the index tree, further increasing access latency; third, the fixed structure design makes it impossible to perceive data distribution characteristics, resulting in low space utilization and difficulty in adapting to the storage needs of massive string keys. Learning indexes, by fitting the cumulative distribution function (CDF) of the data, replace traditional structured traversal with model inference, achieving "computation for memory access," and exhibit excellent performance in numeric keypad scenarios. However, when dealing with variable-length string keys, three major bottlenecks exist: First, string mapping is difficult. The CDF of variable-length strings is step-like, and traditional fixed-length segmentation or prefix truncation methods lose string suffix information and destroy the contextual relationships between characters, leading to high model fitting difficulty and prediction errors. Second, dynamic updates suffer severe degradation. Frequent insertion operations cause data distribution drift, and the model prediction error accumulates continuously, resulting in a sharp decline in query performance. Third, reconstruction blocking defects are prominent. Existing synchronous retraining and state monitoring mechanisms are costly, and in high-concurrency read / write scenarios, performance jitter and expansion blocking problems occur, seriously affecting system stability. Therefore, querying data using traditional indexes or learning indexes is inefficient and cannot meet the core requirements of high-performance in-memory databases and key-value storage systems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application aims to provide a collaborative control method based on a dynamically evolved hybrid learning index structure. This method addresses the problems of low query efficiency caused by high overhead in string comparison operations, long common prefixes, inability to perceive data distribution characteristics, difficulties in string mapping, severe degradation in dynamic updates, data distribution drift due to frequent insertion operations, and prominent reconstruction blocking defects.

[0005] To achieve the above objectives, in a first aspect, this application provides a collaborative control method based on a dynamically evolved hybrid learning index structure, comprising: The multi-granularity semantic fusion mapping module converts the input variable-length string set into integer values ​​of the target bits, and constructs target hybrid indexes in batches based on the integer values ​​of the target bits. Upon receiving a variable-length string query request, the target integer value corresponding to the variable-length string query request is determined, and the slot corresponding to the target integer value is predicted based on the root learning index node in the target hybrid index. Based on the slot, the target hybrid index is traversed using a recursive descent method. The target node position is located based on the traversal result, and a data query task is executed at the target node position.

[0006] In one embodiment, the step of the multi-granularity semantic fusion mapping module converting the input variable-length string set into integer values ​​of the target bits and constructing target hybrid indexes in batches based on the integer values ​​of the target bits includes: When a variable-length string set is received as input, the global character prior, positional conditional distribution, and first-order Markov transition probability are statistically analyzed from the variable-length string set. Based on the hybrid expert model, the global character prior, the positional conditional distribution, and the first-order Markov transition probability are fused according to preset weights to obtain the conditional probability of each positional character, and a probability prefix tree is constructed based on the conditional probability and the string prefix. On the probability prefix tree, the cumulative probability of the string is calculated by recursive interval shrinkage, and the cumulative probability is converted into an integer value of the target position. The integer value of the target position is then mapped in order-preserving manner. The sorted integer values ​​are segmented based on the minimum conflict degree segmentation strategy, and the corresponding hybrid index nodes are initialized according to the data size of each segment to build the target hybrid index in batches.

[0007] In one embodiment, the step of initializing corresponding hybrid index nodes according to the data size of each segment to build the target hybrid index in batches includes: Based on the data size and conflict level of each data segment, the corresponding hybrid index nodes are initialized in a preset order. The hybrid index nodes include: learning index nodes, compressed array nodes, and prefix subtree nodes. When the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation of the learning index node, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered to rebuild the degraded learning index node into a new learning index node, and the step of batch building the target hybrid index continues.

[0008] In one embodiment, the step of triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation in the learning index node includes: Without introducing a shared counter, a lock-free computing method is used to estimate the node load parameters based on the node construction time, data insertion rate, and data growth rate of the learning index node. Based on the lightweight lock-free degradation perception model in the dynamic evolution framework, degradation detection is performed on the learning index node according to the node load parameters. The degradation detection result is determined based on the Bernoulli experiment strategy. When the determination result is that the learning index node has degraded, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered.

[0009] In one embodiment, after the step of triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation in the learning index node, the method further includes: When the determination result is that the learning index node has deteriorated, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered, and the deteriorated learning index node is marked as read-only through the triggered asynchronous shadow retraining mechanism, and the foreground write operation is directed to the pre-allocated shadow buffer. The background thread is controlled to asynchronously scan the full data of the degraded learning index node, and a new learning index node is reconstructed based on the full data. The incremental data in the shadow buffer is merged into the new learning index node. By performing a target atomic operation, the foreground access pointer is switched from the degraded learning index node to a new learning index node, and the memory resources of the degraded learning index node are reclaimed with delay.

[0010] In one embodiment, after the steps of traversing the target hybrid index using a recursive descent method based on the slot, locating the target node position based on the traversal result, and performing a data query task at the target node position, the method further includes: When the execution result of the data query task meets the preset requirements, the target hybrid index is updated based on the execution result and the target integer value; Multi-dimensional concurrency control is implemented based on fine-grained concurrency control strategies. These multi-dimensional concurrency control strategies include: slot-level comparison and swap operations, container optimistic locking, and transitional dual-zone reads.

[0011] Secondly, this application provides a collaborative control system based on a dynamically evolved hybrid learning index structure, comprising: The construction module is used to convert the input variable-length string set into the integer value of the target bit based on the multi-granularity semantic fusion mapping module, and to build the target hybrid index in batches according to the integer value of the target bit; The prediction module is used to determine the target integer value corresponding to the variable-length string query request when a variable-length string query request is received, and predict the slot corresponding to the target integer value based on the root learning index node in the target hybrid index. The execution module is used to traverse the target hybrid index using a recursive descent method based on the slot, locate the target node position based on the traversal result, and execute a data query task at the target node position.

[0012] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0013] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0014] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0015] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) This application introduces a dynamic evolution hybrid learning index structure including a multi-granularity semantic fusion mapping module, a hybrid index node layer and a dynamic evolution framework. The multi-granularity semantic fusion string order-preserving mapping module serves as a pre-processing module for the index, which is used to convert the input variable-length string set into the integer value of the target position, solve the problem of model fitting difficulty caused by the ladder-like distribution of variable-length strings, realize uniform and order-preserving mapping from strings to values, greatly reduce the fitting difficulty of the learning model, reduce the number of fitting segments, and improve the model prediction accuracy. The hybrid index node layer serves as the core storage and routing module of the index, which consists of learning index nodes, compressed array nodes and prefix subtree nodes. It supports dynamic switching of node forms according to data scale and conflict degree. It can use its root learning index node to predict the slot corresponding to the target integer value and traverse it in a recursive descent manner to perform data query tasks at the target node position, thereby effectively improving the efficiency of querying data and meeting the core requirements of high-performance memory databases and key-value storage systems.

[0016] (2) The dynamic evolution framework in this application serves as the performance guarantee module for the index, including a lightweight lock-free degradation perception model and an asynchronous shadow retraining mechanism. The lightweight lock-free degradation perception model is used to detect whether the learning index node has degraded. If so, the asynchronous shadow retraining mechanism is introduced and triggered to rebuild the degraded learning index node into a new learning index node. This completely decouples the front-end read / write from the back-end reconstruction, thoroughly eliminating the blocking problem caused by reconstruction and expansion, and ensuring performance stability in high-concurrency scenarios. Furthermore, the hybrid index node layer in this application adaptively switches according to data characteristics, suppressing index structure expansion, and its memory usage is significantly lower than that of traditional indexes and existing string learning indexes. In addition, this application also sets up a fine-grained concurrency control strategy to complete slot-level comparison and swap (CAS) operations, container optimistic locking, and transitional dual-zone reading (simultaneously supporting read operations of old and new nodes), avoiding coarse-grained lock contention and ensuring consistency and performance stability in high-concurrency scenarios.

[0017] In summary, this application uses a multi-granularity semantic fusion mapping module to convert the input variable-length string set into integer values ​​for the target position, and then constructs a target hybrid index in batches based on these integer values. Upon receiving a variable-length string query request, it determines the target integer value corresponding to the query request and predicts the slot corresponding to the target integer value based on the root learning index node in the target hybrid index. Based on the slot, it traverses the target hybrid index using a recursive descent method, locates the target node position based on the traversal results, and executes a data query task at the target node position. By introducing a multi-granularity semantic fusion mapping module, variable-length strings with a stepped distribution are converted into uniformly linear integer values ​​while strictly preserving the lexicographical order of the strings. This process constructs a target hybrid index, predicts slots, locates target node positions, and queries data, effectively improving the efficiency of data querying and meeting the core requirements of high-performance in-memory databases and key-value storage systems. Attached Figure Description

[0018] Figure 1 This is one of the flowcharts illustrating the collaborative control method based on a dynamically evolved hybrid learning index structure provided in this application embodiment; Figure 2 This is a schematic diagram of the dynamic evolution hybrid learning index structure provided in the embodiments of this application; Figure 3 This is the second flowchart illustrating the collaborative control method based on a dynamically evolved hybrid learning index structure provided in this application embodiment; Figure 4 This is a schematic diagram of the trigger probability change curve provided in the embodiments of this application; Figure 5 This is a schematic diagram of the processing flow of the asynchronous shadow retraining mechanism provided in the embodiments of this application; Figure 6 This is a schematic diagram of the module structure of the collaborative control system based on a dynamic evolution hybrid learning index structure provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0021] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0022] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0023] Based on this, embodiments of this application provide a collaborative control method based on a dynamically evolved hybrid learning index structure, referring to... Figure 1 , Figure 1 This is one of the flowcharts illustrating the collaborative control method based on a dynamically evolved hybrid learning index structure provided in this application. In this embodiment, the collaborative control method based on a dynamically evolved hybrid learning index structure includes steps S10 to S30: Step S10: Based on the multi-granularity semantic fusion mapping module, the input variable-length string set is converted into an integer value of the target bit, and the target hybrid index is constructed in batches according to the integer value of the target bit.

[0024] It should be noted that the experimental hardware in this embodiment is: Intel Xeon Gold 6248 CPU, 128GB of memory, and NVMe SSD hard drive; the experimental software is: Ubuntu 20.04 operating system, C++ programming language, and GCC 9.4.0 compiler; the comparison baselines are: ART, HOT, SIndex, RSS, SLIPP, and LITS; the datasets are: wiki, dna, and rand (variable-length strings covering different distribution characteristics).

[0025] It is understood that the dynamically evolved hybrid learning index structure (EvoIndex) in this embodiment uses a three-layer decoupled architecture, as referenced... Figure 2 , Figure 2This is a structural diagram of a dynamically evolving hybrid learning index structure, including: a multi-granularity semantic fusion mapping module, a hybrid index node layer, and a dynamic evolution framework. The multi-granularity semantic fusion string order-preserving mapping module serves as a pre-processing module for the index, converting variable-length strings into uniform, ordered, and smooth integer values, providing highly adaptable input for the subsequent target hybrid index. The hybrid index node layer, as the core storage and routing module of the index, consists of learning index nodes, compressed array nodes, and prefix subtree nodes, supporting dynamic switching of node forms based on data scale and conflict severity, balancing access efficiency and conflict handling capabilities. The dynamic evolution framework, as the performance guarantee module of the index, includes a lightweight, lock-free degradation awareness model and an asynchronous shadow retraining mechanism, enabling accurate identification of node degradation and non-blocking background reconstruction, ensuring performance stability in high-concurrency scenarios.

[0026] It should be understood that the learning index node consists of a linear model and a fixed-length slot array. The linear model is used to quickly predict the slot corresponding to the string mapping value, and the fixed-length slot array is used to store child node pointers or data. Its core function is fast routing, reducing the height of the index tree and memory access overhead. The compressed array node adopts a fixed-length slot design, combined with AVX2 vectorized batch filtering technology, which is suitable for handling small-scale conflict data. It supports two storage forms: inline short keys (directly storing short strings) and external pointers (storing the address of long strings), improving cache friendliness and data access efficiency. The prefix subtree node adopts a highly optimized prefix tree structure, which is suitable for handling large-scale conflicts and long common prefix strings, and can stably support data updates and range query operations. A multi-form node adaptive switching mechanism is adopted to balance routing efficiency, conflict handling capabilities, and memory utilization, suppressing index structure expansion, and the memory usage is significantly lower than that of traditional indexes and existing string learning indexes. The conversion rules for the three node forms are as follows: the node form automatically upgrades as the amount of data and the number of conflicts increase, following the path: empty slot → single data pointer → compressed array node → prefix subtree node. When degradation of the learning index node is detected, an asynchronous retraining mechanism is triggered to rebuild it into a new learning index node. This shortens the query path, improves cache friendliness, and reduces query latency.

[0027] Understandably, upon receiving a set of variable-length input strings, a multi-granularity semantic fusion mapping module is introduced to convert the set of variable-length strings into an integer value representing the target string. This target string can be 64 bits, and the value strictly follows lexicographical order and is evenly distributed with the mapping values ​​of other strings. This ensures stable performance for long keys ranging from 8 to 64 bytes, addressing the performance sensitivity issue of long keys in existing solutions. The batch construction of the target hybrid index is completed by initializing corresponding hybrid index nodes.

[0028] Step S20: Upon receiving a variable-length string query request, determine the target integer value corresponding to the variable-length string query request, and predict the slot corresponding to the target integer value based on the root learning index node in the target hybrid index.

[0029] It should be understood that upon receiving a variable-length string query request, it indicates the entry into the index structure query phase. At this point, the multi-granularity semantic fusion mapping module is introduced again to convert the variable-length string corresponding to the query request into the target integer value. The root learning index node refers to the learning index node located at the root of the root learning index nodes. (Continue to refer to...) Figure 2 The root learning index node can be the learning index node in the first row, and the slot corresponding to the target integer value is predicted based on the root learning index node.

[0030] Step S30: Based on the slot, the target hybrid index is traversed using a recursive descent method. The target node position is located based on the traversal result, and a data query task is executed at the target node position.

[0031] It should be understood that the target node position refers to the node position of the variable-length string in the target hybrid index. To predict the slot corresponding to the target integer value, a recursive descent method is used to traverse the target hybrid index and locate the target node position, achieving precise node location. After executing a data query task at the target node position, the query result is returned. The data query task includes point queries, range queries, and prefix matching queries. This low-latency, high-throughput, low-memory-consumption, non-blocking update, and key-length-insensitive dynamically evolving hybrid learning index structure can adapt to various high-frequency access scenarios for variable-length string keys.

[0032] Furthermore, after step S30, the method further includes: when the execution result of the data query task meets the preset requirements, updating the target hybrid index according to the execution result and the target integer value; performing multi-dimensional concurrency control based on a fine-grained concurrency control strategy, wherein the multi-dimensional concurrency control includes: slot-level comparison and swap operation, container optimistic locking, and transition period dual-zone reading.

[0033] It is understood that this embodiment also includes an update phase, which updates the target hybrid index. The entire update phase is similar to the construction phase. The updated target hybrid index also supports high-concurrency queries, deletions, and updates. To avoid coarse-grained lock contention and ensure consistency and performance stability in high-concurrency scenarios, this embodiment also sets fine-grained concurrency control strategies to complete slot-level compare-and-swap (CAS) operations, container optimistic locking, and transitional dual-zone reads (simultaneously supporting read operations on old and new nodes).

[0034] This embodiment uses a multi-granularity semantic fusion mapping module to convert the input variable-length string set into integer values ​​for the target position, and then constructs a target hybrid index in batches based on these integer values. Upon receiving a variable-length string query request, it determines the target integer value corresponding to the query request and predicts the slot corresponding to the target integer value based on the root learning index node in the target hybrid index. Based on the slot, it traverses the target hybrid index using a recursive descent method, locates the target node position based on the traversal results, and executes a data query task at the target node position. By introducing a multi-granularity semantic fusion mapping module, variable-length strings with a stepped distribution are converted into uniform linear integer values ​​while strictly preserving the lexicographical order of the strings. This process constructs a target hybrid index, predicts slots, locates target node positions, and queries data, effectively improving the efficiency of data querying and meeting the core requirements of high-performance in-memory databases and key-value storage systems.

[0035] In one specific implementation, this application provides steps for constructing a target hybrid index. Please refer to... Figure 3 , Figure 3 This is the second flowchart illustrating the collaborative control method based on a dynamically evolving hybrid learning index structure provided in this application. Step S10 includes steps S101 to S104: Step S101: When receiving the input set of variable-length strings, calculate the global character prior, positional condition distribution, and first-order Markov transition probability from the set of variable-length strings.

[0036] It should be understood that the global character prior represents the overall skewness of the character distribution, the positional conditional distribution represents the pattern of character occurrence at different positions, and the first-order Markov context transition probability represents the dependency relationship between adjacent characters. The global character prior, positional conditional distribution, and first-order Markov transition probability are three types of key statistical information. Taking the variable-length string "S=cad" as an example, the global character prior is represented by P(c), the positional conditional distribution is represented by P(c|k), and the first-order Markov transition probability is represented by P(c|prev), where k represents the character position.

[0037] Step S102: Based on the hybrid expert model, the global character prior, the positional conditional distribution, and the first-order Markov transition probability are fused according to preset weights to obtain the conditional probability of each positional character, and a probability prefix tree is constructed based on the conditional probability and the string prefix.

[0038] Understandably, after statistically determining the global character prior, positional conditional distribution, and first-order Markov transition probabilities, the Hybrid Expert (MoE) model is used to fuse these probabilities according to preset weights. After fusing the conditional probabilities for each character at each position, a probabilistic prefix tree is constructed. Each node in the probabilistic prefix tree corresponds to a prefix of the string, and the edge weights correspond to the conditional probabilities of the characters. Furthermore, the mathematical expression of the MoE model can be represented as: .

[0039] Where λ1 represents the fusion weight of the global character prior, λ2 represents the fusion weight of the positional conditional distribution, λ3 represents the fusion weight of the first-order Markov transition probability, and λ1+λ2+λ3=1, P(c) represents the global character prior, P(c|k) represents the positional conditional distribution, P(c|prev) represents the first-order Markov transition probability, and M represents the context state of the k-th character.

[0040] Step S103: On the probability prefix tree, the cumulative probability of the string is calculated by recursive interval shrinkage, and the cumulative probability is converted into an integer value of the target position. The integer value of the target position is then mapped in order-preserving manner.

[0041] It should be noted that after constructing the probability prefix tree, the cumulative probability of the string is calculated starting from the root node by recursively shrinking the interval. The cumulative probability is then converted into an integer value of the target position by using the interval shrinking method, thereby realizing the transformation from a step-like distribution to a uniform linear distribution.

[0042] It should be understood that this embodiment also includes an order-preserving mapping mechanism to perform order-preserving mapping on the integer values ​​of the target bits. During the mapping process, the lexicographical order of the strings is strictly followed, and the mapping values ​​with other strings are evenly distributed. Fixed-point arithmetic is used to avoid floating-point errors, and an early stopping mechanism is triggered for scenarios with extremely long common prefixes to reduce computational overhead. Through the above methods, the model fitting difficulty caused by the stepped distribution of variable-length strings can be solved, achieving a uniform and order-preserving mapping from strings to numerical values.

[0043] Step S104: The ordered integer value is segmented based on the minimum conflict degree segmentation strategy, and the corresponding hybrid index node is initialized according to the data size of each segment to build the target hybrid index in batches.

[0044] Understandably, the minimum conflict degree segmentation strategy refers to a segmentation strategy based on the minimum conflict degree. After obtaining the mapped ordered integer value, the ordered integer value is segmented based on the minimum conflict degree (FMCD) segmentation strategy. According to the data size of each segment, the corresponding hybrid index node is initialized to complete the batch construction of the target hybrid index.

[0045] Furthermore, the step of initializing corresponding hybrid index nodes according to the data scale of each segment to build the target hybrid index in batches includes: initializing corresponding hybrid index nodes in a preset order according to the data scale and conflict degree of each data segment, wherein the hybrid index nodes include: learning index nodes, compressed array nodes, and prefix subtree nodes; when the lightweight lock-free degradation perception model in the dynamic evolution framework detects that the learning index node has degraded, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered to rebuild the degraded learning index node into a new learning index node, and the step of building the target hybrid index in batches continues.

[0046] It should be noted that for hybrid index nodes, including learning index nodes, compressed array nodes, and prefix subtree nodes, these nodes can dynamically switch forms based on the amount of data and the amount of conflicts. Specifically: when there is no data in the slot, it is an empty slot; when the first data is inserted, the empty slot is converted into a single data pointer, directly pointing to that data; when 2 to 4 data items are inserted and small-scale conflicts occur, the single data pointer is converted into a compressed array node, using AVX2 vectorized scanning to achieve fast data retrieval; when more than 4 data items are inserted, the amount of conflicts is large, or there is a long common prefix, the compressed array node is converted into a HOT prefix subtree node to stably handle conflicts and range queries.

[0047] It should be understood that the dynamic evolution framework includes a lightweight, lock-free degradation awareness model and an asynchronous shadow retraining mechanism. When degradation of a learning index node is detected by the lightweight, lock-free degradation awareness model, the asynchronous shadow retraining mechanism is introduced and triggered to reconstruct the degraded learning index node into a new learning index node. This completely decouples the front-end read / write operations from the back-end reconstruction, thoroughly eliminating the blocking issues caused by reconstruction and expansion, and ensuring performance stability in high-concurrency scenarios. The asynchronous shadow retraining mechanism can be designed using the RCU (Read-Copy-Update) concept.

[0048] Furthermore, the step of triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation of the learning index node includes: estimating node load parameters based on the node construction time, data insertion rate, and data growth ratio of the learning index node using a lock-free computation method without introducing a shared counter; performing degradation detection on the learning index node based on the node load parameters using the lightweight lock-free degradation perception model in the dynamic evolution framework; and determining the degradation detection result based on the Bernoulli experiment strategy, triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the determination result indicates that the learning index node has degraded.

[0049] Understandably, taking the learning index node as an example, to avoid coarse-grained lock contention, this embodiment employs a lock-free computation method without introducing a shared counter. It estimates the node load parameters based on the learning index node's node construction time, data insertion rate, and data growth ratio. These node load parameters are then input into a Sigmoid function, converting the node degradation level into a trigger probability between 0 and 1. A higher trigger probability indicates more severe node degradation; for example, a trigger probability P = 0.85. (Reference) Figure 4 , Figure 4 The diagram illustrates the probability change curves, specifically: In the initial stage, there is ample node space and a low probability of degradation. In the middle stage, the estimated node size reaches the node threshold, increasing the probability of conflict and causing a sharp rise in the probability of degradation. In the later stage, nodes are older, but new insertions still occur, and the probability of degradation approaches 1. Experimental results show that under a 32-thread mixed read / write load, the dynamic evolution hybrid learning index structure in this embodiment improves throughput by 35.16%~66.32% compared to mainstream string learning indexes, significantly reduces P99 latency, and outperforms all existing string indexing schemes in overall performance.

[0050] It should be noted that after performing degradation detection on the learning index node, a Bernoulli experiment strategy needs to be introduced. Specifically, a Bernoulli experiment is performed to determine the degradation detection result: if the generated value between 0 and 1 is less than P, the learning index node is determined to have degraded, triggering the asynchronous shadow retraining mechanism; otherwise, no degradation has occurred, and the node status continues to be monitored. The entire determination process described above does not block front-end query operations, resulting in extremely low computational overhead.

[0051] Furthermore, after the step of triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation of the learning index node, the method further includes: when the determination result is that the learning index node has degraded, triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework, and marking the degraded learning index node as read-only through the triggered asynchronous shadow retraining mechanism, controlling the foreground write operation to be directed to the pre-allocated shadow buffer; controlling the background thread to asynchronously scan the full data of the degraded learning index node, and reconstructing a new learning index node based on the full data, merging the incremental data in the shadow buffer into the new learning index node; and switching the foreground access pointer from the degraded learning index node to the new learning index node through a target atomic operation, and delaying the reclamation of the memory resources of the degraded learning index node.

[0052] It should also be emphasized that when the judgment result indicates that the learning index node has deteriorated, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered, as shown in the reference. Figure 5 The specific operations are as follows: Mark the degraded learning index node as read-only, prohibiting foreground write operations; redirect all data written to the learning index node by the foreground to the shadow buffer; control the background thread to asynchronously scan the full data of the degraded learning index node, rebuild a new learning index node, rebuild the hybrid learning index structure according to the data scale and conflict situation, merge the incremental data in the shadow buffer into the new learning index node to ensure data integrity and consistency; switch the foreground access pointer from the degraded learning index node (old node) to the new learning index node (new node) through CAS atomic operations. The switching process is extremely short and does not affect foreground read and write operations; and after all read operations accessing the old node are completed, delay the reclamation of the memory resources of the degraded learning index node based on the epoch mechanism, completing the entire retraining process.

[0053] In this embodiment, upon receiving an input set of variable-length strings, the global character prior, positional conditional distribution, and first-order Markov transition probabilities are statistically analyzed from the set. Based on a hybrid expert model, the global character prior, positional conditional distribution, and first-order Markov transition probabilities are fused according to preset weights to obtain the conditional probabilities of characters at each position. A probability prefix tree is then constructed based on the conditional probabilities and string prefixes. On the probability prefix tree, the cumulative probability of the string is calculated using a recursive interval shrinking method, and the cumulative probability is converted into an integer value for the target position. The integer value for the target position is then mapped to maintain its order. The ordered integer value is segmented based on a minimum conflict degree segmentation strategy, and corresponding hybrid index nodes are initialized according to the data size of each segment to build the target hybrid index in batches. By using the above method, after statistically analyzing the global character priors, positional conditional distributions, and first-order Markov transition probabilities from a variable-length string set, a hybrid expert model is introduced. This model fuses the conditional probabilities of characters at each position according to preset weights, and performs order-preserving mapping on the integer values ​​of the target position. This transforms the stepped distribution into a uniform linear distribution, solving the model fitting difficulty problem caused by the stepped distribution of variable-length strings. Then, a minimum conflict degree segmentation strategy is introduced to segment the order-preserving integer values, and the target hybrid index is constructed by initializing the corresponding hybrid index nodes. This effectively improves the accuracy of constructing the target hybrid index.

[0054] The following describes the cooperative control system based on a dynamically evolved hybrid learning index structure provided in this application. The cooperative control system described below corresponds to the cooperative control method described above. Please refer to... Figure 6 , Figure 6 This is a schematic diagram of the module structure of a collaborative control system based on a dynamic evolution hybrid learning index structure provided in an embodiment of this application, including: The construction module T10 is used to convert the input variable-length string set into the integer value of the target bit based on the multi-granularity semantic fusion mapping module, and to build the target hybrid index in batches according to the integer value of the target bit.

[0055] The prediction module T20 is used to determine the target integer value corresponding to the variable-length string query request when a variable-length string query request is received, and predict the slot corresponding to the target integer value based on the root learning index node in the target hybrid index.

[0056] The execution module T30 is used to traverse the target hybrid index using a recursive descent method according to the slot, locate the target node position according to the traversal result, and execute a data query task at the target node position.

[0057] This embodiment uses a multi-granularity semantic fusion mapping module to convert the input variable-length string set into integer values ​​for the target position, and then constructs a target hybrid index in batches based on these integer values. Upon receiving a variable-length string query request, it determines the target integer value corresponding to the query request and predicts the slot corresponding to the target integer value based on the root learning index node in the target hybrid index. Based on the slot, it traverses the target hybrid index using a recursive descent method, locates the target node position based on the traversal results, and executes a data query task at the target node position. By introducing a multi-granularity semantic fusion mapping module, variable-length strings with a stepped distribution are converted into uniform linear integer values ​​while strictly preserving the lexicographical order of the strings. This process constructs a target hybrid index, predicts slots, locates target node positions, and queries data, effectively improving the efficiency of data querying and meeting the core requirements of high-performance in-memory databases and key-value storage systems.

[0058] It is understood that the detailed functional implementation of each of the above modules can be found in the description of the aforementioned method embodiments, and will not be repeated here.

[0059] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0060] Based on the methods in the above embodiments, this application provides an electronic device, please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0061] It should be noted that the system may include: a processor 10, a communications interface 20, a memory 30, and a communication bus 40. The processor 10, communications interface 20, and memory 30 communicate with each other via the communication bus 40. The processor 10 can invoke logical instructions stored in the memory 30 to execute the methods described in the above embodiments.

[0062] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product.

[0063] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0064] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor.

[0065] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A collaborative control method based on a dynamically evolved hybrid learning index structure, characterized in that, include: The multi-granularity semantic fusion mapping module converts the input variable-length string set into integer values ​​of the target bits, and constructs target hybrid indexes in batches based on the integer values ​​of the target bits. Upon receiving a variable-length string query request, the target integer value corresponding to the variable-length string query request is determined, and the slot corresponding to the target integer value is predicted based on the root learning index node in the target hybrid index. Based on the slot, the target hybrid index is traversed using a recursive descent method. The target node position is located based on the traversal result, and a data query task is executed at the target node position.

2. The collaborative control method as described in claim 1, characterized in that, The step of converting the input variable-length string set into an integer value of the target bit based on the multi-granularity semantic fusion mapping module, and constructing the target hybrid index in batches based on the integer value of the target bit, includes: When a variable-length string set is received as input, the global character prior, positional conditional distribution, and first-order Markov transition probability are statistically analyzed from the variable-length string set. Based on the hybrid expert model, the global character prior, the positional conditional distribution, and the first-order Markov transition probability are fused according to preset weights to obtain the conditional probability of each positional character, and a probability prefix tree is constructed based on the conditional probability and the string prefix. On the probability prefix tree, the cumulative probability of the string is calculated by recursive interval shrinkage, and the cumulative probability is converted into an integer value of the target position. The integer value of the target position is then mapped in order-preserving manner. The sorted integer values ​​are segmented based on the minimum conflict degree segmentation strategy, and the corresponding hybrid index nodes are initialized according to the data size of each segment to build the target hybrid index in batches.

3. The cooperative control method as described in claim 2, characterized in that, The step of initializing corresponding hybrid index nodes according to the data size of each segment to build the target hybrid index in batches includes: Based on the data size and conflict level of each data segment, the corresponding hybrid index nodes are initialized in a preset order. The hybrid index nodes include: learning index nodes, compressed array nodes, and prefix subtree nodes. When the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation of the learning index node, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered to rebuild the degraded learning index node into a new learning index node, and the step of batch building the target hybrid index continues.

4. The collaborative control method as described in claim 3, characterized in that, The step of triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation in the learning index node includes: Without introducing a shared counter, a lock-free computing method is used to estimate the node load parameters based on the node construction time, data insertion rate, and data growth rate of the learning index node. Based on the lightweight lock-free degradation perception model in the dynamic evolution framework, degradation detection is performed on the learning index node according to the node load parameters. The degradation detection result is determined based on the Bernoulli experiment strategy. When the determination result is that the learning index node has degraded, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered.

5. The cooperative control method as described in claim 3, characterized in that, After the step of triggering the asynchronous shadow retraining mechanism in the dynamic evolution framework when the lightweight lock-free degradation perception model in the dynamic evolution framework detects degradation in the learning index node, the method further includes: When the determination result is that the learning index node has deteriorated, the asynchronous shadow retraining mechanism in the dynamic evolution framework is triggered, and the deteriorated learning index node is marked as read-only through the triggered asynchronous shadow retraining mechanism, and the foreground write operation is directed to the pre-allocated shadow buffer. The background thread is controlled to asynchronously scan the full data of the degraded learning index node, and a new learning index node is reconstructed based on the full data. The incremental data in the shadow buffer is merged into the new learning index node. By performing a target atomic operation, the foreground access pointer is switched from the degraded learning index node to a new learning index node, and the memory resources of the degraded learning index node are reclaimed with delay.

6. The cooperative control method as described in any one of claims 1 to 5, characterized in that, After the steps of traversing the target hybrid index using a recursive descent method based on the slot, locating the target node position based on the traversal result, and performing a data query task at the target node position, the method further includes: When the execution result of the data query task meets the preset requirements, the target hybrid index is updated based on the execution result and the target integer value; Multi-dimensional concurrency control is implemented based on fine-grained concurrency control strategies. These multi-dimensional concurrency control strategies include: slot-level comparison and swap operations, container optimistic locking, and transitional dual-zone reads.

7. A collaborative control device based on a dynamically evolving hybrid learning index structure, characterized in that, include: The construction module is used to convert the input variable-length string set into the integer value of the target bit based on the multi-granularity semantic fusion mapping module, and to build the target hybrid index in batches according to the integer value of the target bit; The prediction module is used to determine the target integer value corresponding to the variable-length string query request when a variable-length string query request is received, and predict the slot corresponding to the target integer value based on the root learning index node in the target hybrid index. The execution module is used to traverse the target hybrid index using a recursive descent method based on the slot, locate the target node position based on the traversal result, and execute a data query task at the target node position.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-6.

Images