A hierarchical hybrid indexing method and apparatus for memory system address translation

Abstract

The application belongs to the technical field of data processing, and specifically discloses a hierarchical hybrid index method and device for memory system address conversion. Through the application, an address continuity determination threshold is set based on an adaptive segmentation unit; a segment type to which a logical address belongs is determined according to a target hybrid adaptation relationship; when the segment type is a hash segment, the upper guiding layer is controlled to guide the address conversion request; and a physical address is retrieved based on the adaptive bidirectional detection unit and the address storage area positioned based on the lower data layer. Through the above-mentioned mode, the index structure composed of the upper guiding layer, the lower data layer and the adaptive segmentation unit is introduced, the target hybrid adaptation relationship is constructed based on the accurate distinction between continuous addresses and discrete addresses, the shared hash index and the adaptive bidirectional detection unit are adopted for differential retrieval, so that the efficiency of retrieving the physical address can be effectively improved, and the demand of the high-performance memory system for high throughput, low delay and low memory overhead can be met.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and more specifically, relates to a hierarchical hybrid indexing method and apparatus for memory system address translation. Background Technology

[0002] In common memory system address translation scenarios, logical addresses exhibit heterogeneous characteristics, with both continuous and discrete elements coexisting. Traditional single indexes (learned indexes, hash indexes) and simple hybrid indexes suffer from insufficient adaptability and significant performance bottlenecks. Static single learned indexes, limited by fixed model parameters, cannot handle dynamic data updates and struggle to adapt to dynamic changes in address distribution. Dynamic single learned indexes suffer from high overhead due to sparse nodes, low search efficiency, and poor concurrency performance. Furthermore, neither type of learned index can adapt to the heterogeneous distribution of logical addresses, resulting in actual retrieval performance falling short of expectations. Static single hash indexes require a large amount of space to suppress collisions. Redundant slots lead to low memory utilization and cannot meet the memory requirements of massive address mappings. Dynamic single indexes suffer from problems such as expansion blocking and significant physical clustering effects. Furthermore, their randomized mapping mechanism is unable to perceive data distribution, which can disrupt physical storage locality, causing high-frequency cache misses and affecting retrieval efficiency. Simple hybrid indexes lack targeted design for memory system address translation scenarios and cannot achieve deep complementarity between learning indexes and hash indexes. They cannot solve the search bottleneck of learning indexes, nor can they overcome the expansion problem of hash indexes. Moreover, their concurrency performance is insufficient, and they cannot achieve optimal performance across all scenarios under complex loads, making them unsuitable for the core requirements of high-performance memory systems. Therefore, the above methods are inefficient in retrieving physical addresses. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this application aims to provide a hierarchical hybrid indexing method and apparatus for memory system address translation. This addresses the problems of existing technologies, such as inability to handle dynamic data updates, difficulty in adapting to dynamic changes in address distribution, inability to adapt to heterogeneous distribution characteristics where logical addresses are both continuous and discrete, inability to adapt to the memory requirements of massive address mappings, existence of expansion blocking, physical aggregation effects, lack of targeted design for memory system address translation scenarios, and low efficiency in retrieving physical addresses due to insufficient concurrency performance.

[0004] To achieve the above objectives, in a first aspect, this application provides a hierarchical hybrid indexing method for memory system address translation, comprising: Based on the adaptive segmentation unit, an address continuity determination threshold is set according to the actual application scenario of memory system address translation, and a target hybrid adaptation relationship is constructed according to the address continuity determination threshold. The shared hash index is then updated according to the target hybrid adaptation relationship. Upon receiving an address translation request, the logical address corresponding to the address translation request is determined based on the upper-layer guidance layer and the improved learning index, and the segment type to which the logical address belongs is located. When the segment type is a hash segment, the upper-level bootstrapping layer is controlled to guide the address translation request. Based on the lower data layer, the address storage area is located according to the updated shared hash index and shared bucket unit, and the physical address is retrieved according to the address storage area based on the adaptive bidirectional probing unit.

[0005] In one embodiment, the step of setting an address continuity determination threshold based on the actual application scenario of memory system address translation using an adaptive segmentation unit, and constructing a target hybrid adaptation relationship based on the address continuity determination threshold, includes: Real-time acquisition of logical address sets under memory system address translation scenarios, and data cleaning of the logical address sets; Based on statistical analysis strategies, feature mining is performed on the cleaned logical address set, and continuous address clusters and discrete address segments are identified according to address distribution characteristics. Based on the adaptive segmentation unit, the address continuity judgment threshold is set according to the actual application scenario of memory system address translation; Based on the segmentation range, continuous address clusters that meet the address continuity determination threshold are divided into preset precise segments, and discrete address fragments that meet the address continuity determination threshold are divided into preset hash segments. A target hybrid adaptation relationship is constructed based on the preset precise segments and the preset hash segments, and the shared hash index is updated based on the target hybrid adaptation relationship.

[0006] In one embodiment, the steps of dividing contiguous address clusters that satisfy the address continuity determination threshold into preset precise segments based on segment division range, and dividing discrete address fragments that satisfy the address continuity determination threshold into preset hash segments, and constructing a target hybrid adaptation relationship based on the preset precise segments and the preset hash segments, include: Determine the dynamic changes in address distribution based on the address distribution characteristics at different times; When it is determined that the address part features have shifted significantly based on the address distribution dynamic change data, the address continuity determination threshold and segment division range are dynamically adjusted to ensure that the index structure composed of the upper guidance layer, the lower data layer and the adaptive segmentation unit always adapts to the data distribution changes. Based on the adjusted segmentation range, continuous address clusters that meet the adjusted address continuity determination threshold are divided into preset precise segments, and discrete address fragments that meet the adjusted address continuity determination threshold are divided into preset hash segments. A target hybrid adaptation relationship is then constructed based on the preset precise segments and the preset hash segments.

[0007] In one embodiment, the step of determining the logical address corresponding to the address translation request based on the improved learning index according to the upper-layer bootstrapping layer and locating the segment type to which the logical address belongs upon receiving an address translation request includes: Upon receiving an address translation request, the logical address corresponding to the address translation request is determined based on the upper-layer guidance layer and the improved learning index. Based on the improved learning index, the hierarchical skipping thread identifies sparse and invalid nodes during the query path traversal process, and determines the optimized node traversal path by skipping the sparse and invalid nodes. The error range tuning thread based on the improved learning index monitors the changes in the prediction error of the learning model in real time based on the kernel change point detection algorithm, and dynamically adjusts the error limit according to the changes in the prediction error. Based on the improved learning index, a branchless binary search thread or a linear scan thread locates the segment type to which the logical address belongs, according to the optimized node traversal path and the adjusted error limit.

[0008] In one embodiment, before the step of identifying sparse and invalid nodes during query path traversal and determining an optimized node traversal path by skipping the sparse and invalid nodes in the hierarchical skip thread based on the improved learning index, the method further includes: When the application scenario of the upper-layer bootstrapping layer is a high-concurrency scenario, the lock-free concurrency control unit and the concurrent splitting and migration unit of the lower-layer data layer work together; Based on the lock-free concurrency unit, lock-free operations are performed on the multi-threaded upper-layer bootstrapping layer according to the copy-on-write protocol, atomic version verification, and reassembly strategy.

[0009] In one embodiment, the step of locating the address storage region based on the updated shared hash index and shared bucket unit according to the lower data layer, and retrieving the physical address based on the address storage region according to the adaptive bidirectional probing unit, includes: Based on the lower data layer, the address storage area is located according to the updated shared hash index and shared bucket unit; When a hash collision is detected, the target collision location is determined; Within the target storage area, a center diffusion detection is performed bidirectionally from the target collision location based on an adaptive bidirectional detection unit; The free storage slots are located based on the diffusion detection results, and the physical addresses are retrieved based on the free storage slots.

[0010] Secondly, this application provides a hierarchical hybrid indexing device for memory system address translation, comprising: The module is used to set an address continuity determination threshold based on the actual application scenario of memory system address translation, based on the adaptive segmentation unit, and to build a target hybrid adaptation relationship based on the address continuity determination threshold, and update the shared hash index based on the target hybrid adaptation relationship. The determination module is used to determine the logical address corresponding to the address translation request based on the upper-layer guidance layer and the improved learning index when an address translation request is received, and to locate the segment type to which the logical address belongs; The control module is used to control the guidance of the address translation request by the upper-layer bootstrapping layer when the segment type is a hash segment; The retrieval module is used to locate the address storage area based on the updated shared hash index and shared bucket unit of the lower data layer, and to retrieve the physical address based on the address storage area using the adaptive bidirectional probing unit.

[0011] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0012] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0013] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0014] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0015] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) This application introduces an index structure consisting of an upper guidance layer, a lower data layer, and adaptive segmentation units, breaking the existing design pattern of simple superposition of hybrid indexes. It achieves deep collaboration between learning indexes and hash indexes, accurately adapts to the heterogeneous distribution characteristics of logical addresses in memory systems, effectively solves the core problem of insufficient adaptability of existing hybrid indexes, achieves the optimal balance between retrieval performance and space utilization, and adapts to different types of address translation scenarios. For the adaptive segmentation unit, the address continuity judgment threshold is set according to the actual application scenario of address translation in memory systems, and the target hybrid adaptation relationship is constructed according to the address continuity judgment threshold to achieve dynamic matching between the index and the load characteristics and dynamically improve performance.

[0016] (2) In terms of the upper-level guidance layer, this application uses an improved learning index, which consists of a hierarchical skip unit, a hybrid search unit, an error range tuning unit, and a lock-free concurrency unit. Each module is equipped with a corresponding thread, such as a hierarchical skip thread, an error range tuning thread, and a branchless binary search thread. The modules work together to improve the retrieval efficiency and concurrency performance of the guidance layer. This addresses the bottlenecks of traditional learning indexes, such as high overhead of sparse nodes, branch prediction failure, and low search efficiency. At the same time, through lock-free concurrency design, the RCU protocol and atomic version verification strategy are adopted to significantly improve the concurrency throughput of the upper-level guidance layer. This can stably adapt to high-concurrency address translation scenarios and solve the problem of poor concurrency performance of traditional learning indexes.

[0017] (3) In terms of the lower data layer, this application uses a shared hash index, which consists of a shared bucket unit, an adaptive bidirectional probe unit, and a concurrent splitting and migration unit. Each module has a clear division of labor and works together to effectively solve the technical defects of traditional hash indexes, such as low memory utilization, obvious physical aggregation effect, and expansion blocking. It achieves a dual improvement in high memory efficiency and high concurrency stability, and can adapt to the needs of massive address mapping. It has significant improvements in throughput, tail latency, memory overhead, and scalability, supports linear expansion, and can adapt to the application needs of massive address mapping scenarios.

[0018] In summary, this application, based on an adaptive segmentation unit, sets an address continuity determination threshold according to the actual application scenario of address translation in a memory system, constructs a target hybrid adaptation relationship based on the address continuity determination threshold, and updates the shared hash index based on the target hybrid adaptation relationship. Upon receiving an address translation request, based on the upper-layer guidance layer, it determines the logical address corresponding to the address translation request based on an improved learning index and locates the segment type to which the logical address belongs. When the segment type is a hash segment, it controls the upper-layer guidance layer to guide the address translation request. Based on the lower-layer data layer, it locates the address storage area based on the updated shared hash index and shared bucket unit, and retrieves the physical address based on the address storage area using an adaptive bidirectional probing unit. Through this approach, an index structure consisting of an upper-layer guidance layer, a lower-layer data layer, and an adaptive segmentation unit is introduced. Based on accurately distinguishing between continuous and discrete addresses and constructing a target hybrid adaptation relationship, it uses a shared hash index and an adaptive bidirectional probing unit for differentiated retrieval, thereby effectively improving the efficiency of physical address retrieval and meeting the requirements of high-performance memory systems for high throughput, low latency, and low memory overhead. Attached Figure Description

[0019] Figure 1 This is one of the flowcharts illustrating a hierarchical hybrid indexing method for memory system address translation provided in this application embodiment; Figure 2 This is a schematic diagram of the overall index structure provided in the embodiments of this application; Figure 3 This is a schematic diagram of the segmented process provided in the embodiments of this application; Figure 4 This is a schematic diagram of the process of dynamic migration of logical boundaries and in-situ borrowing of slots provided in the embodiments of this application; Figure 5 This is a flowchart of concurrent splitting and smooth migration provided in the embodiments of this application; Figure 6 This is the second flowchart of the hierarchical hybrid indexing method for memory system address translation provided in the embodiments of this application; Figure 7 This is a schematic diagram of the module structure of the hierarchical hybrid indexing device for memory system address translation provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0022] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] Based on this, embodiments of this application provide a hierarchical hybrid indexing method for memory system address translation, referring to... Figure 1 , Figure 1 This is one of the flowcharts illustrating a hierarchical hybrid indexing method for memory system address translation provided in this application embodiment. In this embodiment, the hierarchical hybrid indexing method for memory system address translation includes steps S10 to S40: Step S10: Based on the adaptive segmentation unit, set the address continuity determination threshold according to the actual application scenario of memory system address translation, construct the target hybrid adaptation relationship according to the address continuity determination threshold, and update the shared hash index according to the target hybrid adaptation relationship.

[0025] It should be noted that the reference Figure 2 , Figure 2 The diagram illustrates the overall index structure, which includes an upper-level bootstrapping layer, a lower-level data layer, and an adaptive segmentation unit. The adaptive segmentation unit, as the core component connecting the upper-level bootstrapping layer and the lower-level data layer, can analyze address distribution characteristics in real time, accurately distinguish between continuous and discrete addresses, divide continuous addresses into precise segments, and divide discrete addresses into hash segments. This achieves precise adaptation between the index structure and data distribution characteristics. Precise segments include fields such as start key and continuous address, while hash segments include fields such as start key, length, and continuous address. The overall process revolves around four core stages: address acquisition, feature analysis, segmentation, and dynamic optimization. Each stage is closely connected and operates collaboratively.

[0026] It should be understood that the address continuity determination threshold refers to the threshold used to divide contiguous addresses, connect addresses, and construct the target hybrid adaptation relationship. This address continuity determination threshold can be reasonably set according to the actual application scenario of memory system address translation. After constructing the target hybrid adaptation relationship, the discrete address segments divided into hash segments are determined according to the target hybrid adaptation relationship, and they are allocated to the shared hash index of the lower data layer to realize the update of the shared hash index. (Reference) Figure 3 , Figure 3 The diagram below illustrates the segmentation process: The left side represents the division of precise segments, using the base logical address (BLA) and base physical address (BPA) as examples. The final segmentation result is: mapping entries → BPA, PA, PA, logical addresses → BLA, LA, LA; The right side represents the division of hash segments, using the base logical address (BLA), number of buckets, segment capacity, and base physical address (BPA) as examples. The final segmentation result is: mapping entries → PA, PA, PA, logical addresses LA, LA, LA.

[0027] Step S20: Upon receiving an address translation request, based on the upper-layer guidance layer and the improved learning index, determine the logical address corresponding to the address translation request and locate the segment type to which the logical address belongs.

[0028] Understandably, the upper-level guidance layer uses an improved learning index, whose core function is to quickly locate precise and hash segments, providing efficient retrieval guidance for the lower-level data layer. Upon receiving an address translation request, the improved learning index determines the logical address corresponding to the request and identifies the segment type of that logical address, which can be either a hash segment or a precise segment.

[0029] Further, step S20 includes: upon receiving an address translation request, determining the logical address corresponding to the address translation request based on the upper-layer guidance layer and the improved learning index; identifying sparse invalid nodes during the query path traversal process using the hierarchical skip thread based on the improved learning index, and determining an optimized node traversal path by skipping the sparse invalid nodes; monitoring the prediction error changes of the learning model in real time using the kernel change detection algorithm based on the error range tuning thread based on the improved learning index, and dynamically adjusting the error limit according to the prediction error changes; and locating the segment type to which the logical address belongs based on the optimized node traversal path and the adjusted error limit using the branchless binary search thread or linear scan thread based on the improved learning index.

[0030] It should be understood that the improved learning index is structured as follows: a hierarchical skip unit, a hybrid search unit, an error range tuning unit, and a lock-free concurrency unit. Each module has a corresponding thread, such as a hierarchical skip thread, an error range tuning thread, and a branchless binary search thread. These modules work together to improve the retrieval efficiency and concurrency performance of the guidance layer. Specifically, the output of the hierarchical skip unit is connected to the hybrid search unit; the hybrid search unit and the error range tuning unit interact bidirectionally. The error range tuning unit provides dynamically adjusted error limits to the hybrid search unit, and the hybrid search unit feeds back errors encountered during the search process to the error range tuning unit for threshold calibration. The lock-free concurrency unit covers the entire guidance layer and runs through the operation of each module, ensuring efficient concurrent operation of each module in multi-threaded scenarios and avoiding performance losses caused by lock contention. The functions of each module support each other, forming a complete guidance layer optimization system.

[0031] Understandably, the core functions of each module work together to form the core advantages of the improved learning index. For the hierarchical skipping thread, it addresses the problem of sparse top-level nodes in traditional learning indexes. It incorporates node density-based filtering rules, automatically identifying and skipping sparse and invalid nodes during query path traversal, reducing unnecessary pointer dereference overhead and cold cache latency. This allows the query path to quickly converge to the valid node region, improving node traversal efficiency. For the error range tuning thread, based on a kernel variable detection algorithm, it monitors changes in the learning model's prediction error in real time. By analyzing local features of the data distribution, it dynamically adjusts the error limits, achieving an optimal balance between search accuracy and retrieval efficiency, ensuring the accuracy of the prediction results. When the learning model's prediction error range is small, a CPU-cache-friendly linear scan thread is used for positional location. When the learning model's prediction error range is large, it switches to a branchless binary search thread. This dynamic switching of retrieval threads effectively avoids pipeline stalls caused by branch prediction failures, significantly improving search efficiency.

[0032] Furthermore, before the step of identifying sparse and invalid nodes during the query path traversal process and determining the optimized node traversal path by skipping the sparse and invalid nodes, the method further includes: when the application scenario of the upper-layer bootstrapping layer is a high-concurrency scenario, controlling the lock-free concurrency unit to work in coordination with the concurrent splitting and migration unit of the lower-layer data layer; and based on the lock-free concurrency unit, performing lock-free operations on the multi-threaded system of the upper-layer bootstrapping layer according to the copy-on-write protocol, atomic version verification, and reorganization strategy.

[0033] Understandably, for high-concurrency scenarios in the upper bootstrapping layer, this embodiment also sets up a lock-free concurrency unit to ensure lock-free multi-threaded retrieval, avoiding performance loss caused by lock contention. It works in conjunction with the concurrent splitting and migration unit in the lower data layer to reduce instruction pipeline pauses, improve bootstrapping efficiency in concurrent scenarios, and ensure that the upper bootstrapping layer can adapt to high-concurrency address translation requests. Specifically, based on the lock-free concurrency unit, lock-free operations are performed on the multi-threaded upper bootstrapping layer according to the copy-on-write (RCU) protocol, atomic version verification, and reassembly strategy, avoiding lock contention between threads. Simultaneously, it ensures the consistency of multi-threaded access, thereby significantly improving concurrent throughput and adapting to the needs of high-concurrency address translation scenarios. The concurrent splitting and migration unit ensures that the expansion process does not block address translation requests, jointly guaranteeing the consistency and performance stability of multi-threaded access.

[0034] Step S30: When the segment type is a hash segment, control the upper-layer bootstrapping layer to guide the address translation request.

[0035] It should be understood that after determining the segment type of the logical address, it is necessary to determine whether the segment type is a hash segment. If so, the upper-layer bootstrapping layer is controlled to direct the address translation request, that is, to direct the address translation request to the shared hash index used by the lower-layer data layer. Otherwise, if not, the corresponding physical address can be obtained directly through a simple calculation of "base address + offset", which can quickly complete the address translation without additional retrieval operations and greatly reduce latency.

[0036] Step S40: Based on the lower data layer, locate the address storage area according to the updated shared hash index and shared bucket unit, and retrieve the physical address according to the address storage area based on the adaptive bidirectional probe unit.

[0037] It should be noted that since the adaptive segmentation unit allocates the divided hash segments to the shared hash index, when an address translation request directed by the upper-layer bootstrapping layer is received, a hash retrieval process is triggered. That is, the address storage area is located according to the updated shared hash index and the shared bucket unit, and the physical address is retrieved according to the address storage area based on the adaptive bidirectional probing unit.

[0038] It should be understood that the structure of a shared hash index consists of shared bucket units, adaptive bidirectional probing units, and concurrent splitting and migration units. Each module has a clear division of labor and works collaboratively to address the technical shortcomings of traditional hash indexes, improving retrieval efficiency and stability. The shared bucket unit, as the basic storage unit, provides storage space for address-mapped entries and is the core carrier of the entire hash index. Its design breaks the fixed partitioning pattern of bucket resources in traditional hash indexes. The adaptive bidirectional probing unit is directly connected to the shared bucket unit to handle hash collisions. The concurrent splitting and migration unit interacts with the shared bucket unit in real time, monitoring the load status of the shared bucket unit. When the load reaches a preset threshold, it initiates concurrent splitting and data migration operations to achieve dynamic expansion, ensuring the index's scalability and performance stability.

[0039] Further, step S40 includes: locating the address storage area based on the lower data layer, according to the updated shared hash index and shared bucket unit; determining the target conflict location when a hash conflict is detected; performing a center diffusion probe along both directions from the target conflict location within the target storage area based on the adaptive bidirectional probe unit; locating an idle storage slot based on the diffusion probe result, and retrieving the physical address based on the idle storage slot.

[0040] Understandably, the core functions of each unit revolve around optimizing the performance of hash indexes, forming a complete technical system. For the shared bucket unit, a dynamic migration strategy for logical boundaries is designed to allow multiple hash segments to share bucket resources, enabling in-situ borrowing of storage slots to locate address storage areas, referencing... Figure 4 , Figure 4 The flowchart illustrates the dynamic migration of logical boundaries and in-situ borrowing of slots. Specifically, the blue box represents elements in bucket 1, the green box represents elements in bucket 2, the blank box represents empty slots, B1 represents the starting pointer of bucket 1, and B2 represents the starting pointer of bucket 2. The first step attempts to insert a new element X, triggering a hash collision. The second step involves hash bucket element migration, where element E in bucket 2 moves from the starting pointer to the end. The third step involves boundary movement, successfully inserting element X. This effectively improves memory utilization, reduces the waste of redundant slots, and lowers memory overhead, adapting to the memory requirements of scenarios with massive address mappings. For the adaptive bidirectional probing unit, the traditional unidirectional probing strategy is no longer used; instead, a center-diffusion probing strategy is employed. This involves bidirectionally probing free slots from the target collision location towards both ends and retrieving the physical address based on the free storage slot. This effectively suppresses physical aggregation effects, shortens the probing sequence length, improves collision resolution efficiency, and ensures the stability of retrieval latency.

[0041] It should be noted that the concurrent splitting and migration unit also has a load-aware concurrent splitting and smooth migration mechanism, which can be found in the following reference. Figure 5 , Figure 5 The flowchart for concurrent splitting and smooth migration is as follows: The load factor of each hash segment is monitored in real time. When the load reaches a preset threshold based on the load factor, the concurrent splitting operation is automatically started. The routing check is performed to see if B[idx]==1 is true. If it is, the migration is performed as needed. If not, B[idx]==0 is adopted. A dual-version view coexistence strategy is adopted to retain the old version index and the new version index at the same time. The smooth migration of data is achieved through a background thread to avoid performance blocking during the expansion process, ensure that address translation requests can be processed continuously and efficiently, and ensure the performance stability of the system under high load scenarios.

[0042] This embodiment, based on an adaptive segmentation unit, sets an address continuity determination threshold according to the actual application scenario of address translation in the memory system, constructs a target hybrid adaptation relationship based on the address continuity determination threshold, and updates the shared hash index based on the target hybrid adaptation relationship. Upon receiving an address translation request, based on the upper-layer guidance layer and an improved learning index, the logical address corresponding to the address translation request is determined, and the segment type to which the logical address belongs is located. When the segment type is a hash segment, the upper-layer guidance layer is controlled to guide the address translation request. By introducing an index structure composed of an upper-layer guidance layer, a lower-layer data layer, and an adaptive segmentation unit, and by accurately distinguishing between continuous and discrete addresses, constructing a target hybrid adaptation relationship, and using a shared hash index and an adaptive bidirectional probing unit for differentiated retrieval, the efficiency of physical address retrieval can be effectively improved, meeting the requirements of high-performance memory systems for high throughput, low latency, and low memory overhead.

[0043] In one specific implementation, this application provides steps for constructing a target hybrid adaptation relationship. Please refer to... Figure 6 , Figure 6 This is the second flowchart illustrating the hierarchical hybrid indexing method for memory system address translation provided in this application embodiment. Step S10 includes steps S101 to S104: Step S101: Collect the set of logical addresses in the memory system address translation scenario in real time, and clean the set of logical addresses.

[0044] It should be noted that the adaptive segmentation unit, as a core component connecting the upper bootstrapping layer and the lower data layer, achieves precise adaptation of index and address distribution, i.e., constructs a target hybrid adaptation relationship. After collecting the logical address set in the memory system address translation scenario in real time, the logical address set undergoes data cleaning to eliminate abnormal address interference.

[0045] Step S102: Based on statistical analysis strategies, feature mining is performed on the cleaned logical address set, and continuous address clusters and discrete address segments are identified according to address distribution characteristics.

[0046] Understandably, after cleaning the logical address set, introducing statistical analysis strategies to perform feature mining on the cleaned logical address set and identifying continuous address clusters and discrete address fragments based on address distribution characteristics can effectively improve the accuracy of identification. Continuous address clusters mainly originate from sequential writes of large files, while discrete address fragments mostly come from random writes of massive amounts of fragmented small files.

[0047] Step S103: Based on the adaptive segmentation unit, set the address continuity judgment threshold according to the actual application scenario of memory system address translation.

[0048] Step S104: Divide the continuous address clusters that meet the address continuity determination threshold into preset precise segments according to the segment division range, and divide the discrete address fragments that meet the address continuity determination threshold into preset hash segments. Construct a target hybrid adaptation relationship based on the preset precise segments and the preset hash segments, and update the shared hash index based on the target hybrid adaptation relationship.

[0049] Understandably, after reasonably setting the address continuity judgment threshold according to the actual application scenario of address translation in the memory system, the continuous address clusters that meet the address continuity judgment threshold are divided into preset precise segments, and the discrete address fragments that meet the address continuity judgment threshold are divided into preset hash segments. The precise segments are used to store continuous address clusters, and the hash segments are used to store discrete address fragments. The two-layer structure works together to complete the address translation, and constructs the target hybrid adaptation relationship based on the preset precise segments, preset hash segments, and logical address sets. Among them, the preset hash segments are allocated to the shared hash index used by the lower data layer to complete the update of the shared hash index.

[0050] Further, step S104 includes: determining dynamic change data of address distribution based on address distribution characteristics at different times; when it is determined from the dynamic change data of address distribution that the address part characteristics have shifted significantly, dynamically adjusting the address continuity determination threshold and segment division range to ensure that the index structure composed of the upper guidance layer, the lower data layer, and the adaptive segmentation unit always adapts to the changes in data distribution; dividing the continuous address clusters that meet the adjusted address continuity determination threshold into preset precise segments and the discrete address fragments that meet the adjusted address continuity determination threshold into preset hash segments according to the adjusted segment division range, and constructing a target hybrid adaptation relationship based on the preset precise segments and the preset hash segments.

[0051] It should be understood that during the entire process of building the target hybrid adaptation relationship, the dynamic changes in address distribution will be monitored in real time. That is, the dynamic change data of address distribution will be determined based on the address distribution characteristics at different times. When it is determined that the address part features have shifted significantly based on the dynamic change data of address distribution, the address continuity judgment threshold and segmentation range will be dynamically adjusted to ensure that the index structure composed of the upper guidance layer, the lower data layer and the adaptive segmentation unit always adapts to the changes in data distribution, maintains stable retrieval performance, realizes dynamic matching between index and load characteristics, and dynamically improves performance.

[0052] This embodiment collects a set of logical addresses in a memory system address translation scenario in real time and cleans the logical address set. Based on statistical analysis strategies, it performs feature mining on the cleaned logical address set and identifies continuous address clusters and discrete address segments according to address distribution characteristics. Based on an adaptive segmentation unit, it sets an address continuity judgment threshold according to the actual application scenario of memory system address translation. According to the segmentation range, continuous address clusters that meet the address continuity judgment threshold are divided into preset precise segments, and discrete address segments that meet the address continuity judgment threshold are divided into preset hash segments. A target hybrid adaptation relationship is then constructed based on the preset precise segments and the preset hash segments. Through the above method, after collecting the logical address set in a memory system address translation scenario in real time, the logical address set is cleaned to eliminate abnormal address interference. Continuous address clusters and discrete address segments are accurately identified based on address distribution characteristics. An address continuity judgment threshold is reasonably set according to the actual application scenario of memory system address translation. A target hybrid adaptation relationship is constructed based on the preset precise segments and preset hash segments, achieving accurate adaptation between the index and address distribution.

[0053] The hierarchical hybrid indexing device for memory system address translation provided in this application is described below. The hierarchical hybrid indexing device for memory system address translation described below corresponds to the hierarchical hybrid indexing method for memory system address translation described above. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the module structure of a hierarchical hybrid indexing device for memory system address translation provided in an embodiment of this application, including: The construction module T10 is used to set an address continuity determination threshold based on the actual application scenario of memory system address translation based on the adaptive segmentation unit, construct a target hybrid adaptation relationship based on the address continuity determination threshold, and update the shared hash index based on the target hybrid adaptation relationship.

[0054] The determination module T20 is used to determine the logical address corresponding to the address translation request based on the upper-layer guidance layer and the improved learning index when an address translation request is received, and to locate the segment type to which the logical address belongs.

[0055] The control module T30 is used to control the upper-layer bootstrapping layer to guide the address translation request when the segment type is a hash segment.

[0056] The retrieval module T40 is used to locate the address storage area based on the updated shared hash index and shared bucket unit of the lower data layer, and to retrieve the physical address based on the address storage area using the adaptive bidirectional probing unit.

[0057] This embodiment, based on an adaptive segmentation unit, sets an address continuity threshold according to the actual application scenario of address translation in the memory system, constructs a target hybrid adaptation relationship based on the address continuity threshold, and updates the shared hash index based on the target hybrid adaptation relationship. Upon receiving an address translation request, based on the upper-layer guidance layer, it determines the logical address corresponding to the address translation request based on the improved learning index and locates the segment type to which the logical address belongs. When the segment type is a hash segment, it controls the upper-layer guidance layer to guide the address translation request. Based on the lower-layer data layer, it locates the address storage area based on the updated shared hash index and shared bucket unit, and retrieves the physical address based on the address storage area using an adaptive bidirectional probing unit. Through the above method, an index structure consisting of an upper-layer guidance layer, a lower-layer data layer, and an adaptive segmentation unit is introduced. Based on accurately distinguishing between continuous and discrete addresses, a target hybrid adaptation relationship is constructed, and differentiated retrieval is performed using a shared hash index and an adaptive bidirectional probing unit. This effectively improves the efficiency of physical address retrieval and meets the requirements of high-performance memory systems for high throughput, low latency, and low memory overhead.

[0058] It is understood that the detailed functional implementation of each of the above modules can be found in the description of the aforementioned method embodiments, and will not be repeated here.

[0059] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0060] Based on the methods in the above embodiments, this application provides an electronic device, please refer to... Figure 8 , Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0061] It should be noted that the system may include: a processor 10, a communications interface 20, a memory 30, and a communication bus 40. The processor 10, communications interface 20, and memory 30 communicate with each other via the communication bus 40. The processor 10 can invoke logical instructions stored in the memory 30 to execute the methods described in the above embodiments.

[0062] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0063] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0064] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0065] It is understood that the processor in the embodiments of this application can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0066] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor.

[0067] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A hierarchical hybrid indexing method for memory system address translation, characterized in that, include: Based on the adaptive segmentation unit, an address continuity determination threshold is set according to the actual application scenario of memory system address translation, and a target hybrid adaptation relationship is constructed according to the address continuity determination threshold. The shared hash index is then updated according to the target hybrid adaptation relationship. Upon receiving an address translation request, the logical address corresponding to the address translation request is determined based on the upper-layer guidance layer and the improved learning index, and the segment type to which the logical address belongs is located. When the segment type is a hash segment, the upper-layer bootstrapping layer is controlled to guide the address translation request. Based on the lower data layer, the address storage area is located according to the updated shared hash index and shared bucket unit, and the physical address is retrieved according to the address storage area based on the adaptive bidirectional probing unit.

2. The hierarchical hybrid indexing method as described in claim 1, characterized in that, The steps of setting an address continuity determination threshold based on the actual application scenario of memory system address translation based on the adaptive segmentation unit, constructing a target hybrid adaptation relationship based on the address continuity determination threshold, and updating the shared hash index based on the target hybrid adaptation relationship include: Real-time acquisition of logical address sets under memory system address translation scenarios, and data cleaning of the logical address sets; Based on statistical analysis strategies, feature mining is performed on the cleaned logical address set, and continuous address clusters and discrete address segments are identified according to address distribution characteristics. Based on the adaptive segmentation unit, the address continuity judgment threshold is set according to the actual application scenario of memory system address translation; Based on the segmentation range, continuous address clusters that meet the address continuity determination threshold are divided into preset precise segments, and discrete address fragments that meet the address continuity determination threshold are divided into preset hash segments. A target hybrid adaptation relationship is constructed based on the preset precise segments and the preset hash segments, and the shared hash index is updated based on the target hybrid adaptation relationship.

3. The hierarchical hybrid indexing method as described in claim 2, characterized in that, The steps of dividing contiguous address clusters that meet the address continuity determination threshold into preset precise segments based on segment division range, and dividing discrete address fragments that meet the address continuity determination threshold into preset hash segments, and constructing a target hybrid adaptation relationship based on the preset precise segments and the preset hash segments, include: Determine the dynamic changes in address distribution based on the address distribution characteristics at different times; When it is determined that the address part features have shifted significantly based on the address distribution dynamic change data, the address continuity determination threshold and segment division range are dynamically adjusted to ensure that the index structure composed of the upper guidance layer, the lower data layer and the adaptive segmentation unit always adapts to the data distribution changes. Based on the adjusted segmentation range, continuous address clusters that meet the adjusted address continuity determination threshold are divided into preset precise segments, and discrete address fragments that meet the adjusted address continuity determination threshold are divided into preset hash segments. A target hybrid adaptation relationship is then constructed based on the preset precise segments and the preset hash segments.

4. The hierarchical hybrid indexing method as described in claim 1, characterized in that, The step of determining the logical address corresponding to the address translation request based on the improved learning index and the upper-layer bootstrapping layer, and locating the segment type to which the logical address belongs, upon receiving an address translation request, includes: Upon receiving an address translation request, the logical address corresponding to the address translation request is determined based on the upper-layer guidance layer and the improved learning index. Based on the improved learning index, the hierarchical skipping thread identifies sparse and invalid nodes during the query path traversal process, and determines the optimized node traversal path by skipping the sparse and invalid nodes. The error range tuning thread based on the improved learning index monitors the changes in the prediction error of the learning model in real time based on the kernel change point detection algorithm, and dynamically adjusts the error limit according to the changes in the prediction error. Based on the improved learning index, a branchless binary search thread or a linear scan thread locates the segment type to which the logical address belongs, according to the optimized node traversal path and the adjusted error limit.

5. The hierarchical hybrid indexing method as described in claim 4, characterized in that, Before the step of identifying sparse and invalid nodes during query path traversal and determining the optimized node traversal path by skipping the sparse and invalid nodes, the hierarchical skip thread based on the improved learning index further includes: When the application scenario of the upper-layer bootstrapping layer is a high-concurrency scenario, the lock-free concurrency control unit and the concurrent splitting and migration unit of the lower-layer data layer work together; Based on the lock-free concurrency unit, lock-free operations are performed on the multi-threaded upper-layer bootstrapping layer according to the copy-on-write protocol, atomic version verification, and reassembly strategy.

6. The hierarchical hybrid indexing method as described in any one of claims 1 to 5, characterized in that, The steps of locating the address storage area based on the updated shared hash index and shared bucket unit, and retrieving the physical address based on the address storage area using the adaptive bidirectional probing unit, include: Based on the lower data layer, the address storage area is located according to the updated shared hash index and shared bucket unit; When a hash collision is detected, the target collision location is determined; Within the target storage area, a center diffusion detection is performed bidirectionally from the target collision location based on an adaptive bidirectional detection unit; The free storage slots are located based on the diffusion detection results, and their physical addresses are retrieved based on the free storage slots.

7. A hierarchical hybrid indexing device for memory system address translation, characterized in that, include: The module is used to set an address continuity determination threshold based on the actual application scenario of memory system address translation, based on the adaptive segmentation unit, and to build a target hybrid adaptation relationship based on the address continuity determination threshold, and update the shared hash index based on the target hybrid adaptation relationship. The determination module is used to determine the logical address corresponding to the address translation request based on the upper-layer guidance layer and the improved learning index when an address translation request is received, and to locate the segment type to which the logical address belongs; The control module is used to control the guidance of the address translation request by the upper-layer bootstrapping layer when the segment type is a hash segment; The retrieval module is used to locate the address storage area based on the updated shared hash index and shared bucket unit of the lower data layer, and to retrieve the physical address based on the address storage area using the adaptive bidirectional probing unit.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-6.

Images