A hierarchical adaptive caching method for large-scale sampling spaces

Abstract

This invention discloses a hierarchical adaptive caching method for large-scale sampling spaces, belonging to the field of data processing. The method includes: initializing the storage space and nodes of the sampling system to construct a storage operation model for data sampling and enumeration retrieval; performing hierarchical mapping on externally input sampling requests according to the storage operation model, and performing real-time simplification processing on the sampling path nodes that have completed the hierarchical mapping to obtain accurate positioning results of the target leaf nodes and a simplified node set retaining only node navigation information; and performing adaptive scheduling processing on the simplified node set to obtain a hierarchical adaptive cache set that meets resource and access requirements. This invention can improve response speed and memory utilization during large-scale spatial sampling.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a hierarchical adaptive caching method for a large-scale sampling space. Background Technology

[0002] In tasks such as massive data analysis, scientific computing, and large-scale graph mining, it is often necessary to sample or enumerate a huge implicit target space. When calculating related results, a logical sampling tree is typically constructed. This tree recursively divides the huge target space into multiple logical sub-partitions and establishes a one-to-one mapping between elements in the implicit space and leaf nodes of the sampling tree. By performing top-down weight mapping on the sampling tree, the system can locate a specific tuple result from a randomly generated integer seed, thus achieving random access.

[0003] In related technologies, when sampling ultra-large-scale implicit spaces, the lack of an adaptive trade-off mechanism between computational overhead and storage resources can easily lead to high response latency and memory resource exhaustion, thus failing to meet the high throughput and high stability requirements of large-scale sampling enumeration in scenarios such as big data analysis and machine learning.

[0004] Therefore, there is an urgent need for a hierarchical adaptive caching method for large-scale sampling spaces to solve the above-mentioned technical problems. Summary of the Invention

[0005] This invention provides a hierarchical adaptive caching method for large-scale sampling spaces, which can improve response speed and memory utilization when performing large-scale spatial sampling. The technical solution is as follows: On the one hand, a hierarchical adaptive caching method for a large-scale sampling space is provided, the method comprising: The storage space and nodes of the sampling system are initialized to construct a storage and operation model for data sampling and enumeration retrieval. According to the storage and operation model, the sampling requests of external inputs are mapped in layers, and the sampling path nodes that have completed the layer mapping are simplified in real time to obtain the accurate positioning result of the target leaf node and the simplified node set that retains only the node navigation information. The simplified node set is subjected to adaptive scheduling to obtain a hierarchical adaptive cache set that meets resource and access requirements.

[0006] On the other hand, a hierarchical adaptive caching device for a large-scale sampling space is provided, the device comprising: The initialization module is used to initialize the storage space and nodes of the sampling system and build a storage and operation model for data sampling and enumeration retrieval. The sampling module is used to perform hierarchical mapping of sampling requests from external inputs according to the storage and operation model, and to perform real-time simplification of the sampling path nodes that have completed the hierarchical mapping, so as to obtain the accurate positioning result of the target leaf node and a simplified node set that retains only the node navigation information. The scheduling module is used to perform adaptive scheduling processing on the simplified node set to obtain a hierarchical adaptive cache set that meets resource and access requirements.

[0007] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for executing the computer program stored in the memory to implement the steps of the hierarchical adaptive caching method for large-scale sampling space described above.

[0008] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the steps of the hierarchical adaptive caching method for the large-scale sampling space described above are implemented.

[0009] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the hierarchical adaptive caching method for the large-scale sampling space described above.

[0010] The technical solution provided by this invention can bring at least the following beneficial effects: First, this method constructs a storage operation model by dividing a global cache pool and defining node states. Then, it uses this model to perform hierarchical and rapid retrieval of sampling requests. Simultaneously, after each layer of retrieval, an immediate thinning and release strategy is executed. After the child node's calculation is completed, the parent node's details are immediately erased, dynamically compressing memory usage to the current sampling depth, fundamentally preventing unlimited memory expansion. Finally, based on adaptive management of access frequency, high-frequency nodes are intelligently retained, while low-frequency nodes are eliminated, maximizing the cache hit rate within limited memory. Overall, this solution, through a collaborative mechanism of "materialization on demand, dismantling after use, and retention of hotspots," significantly reduces memory usage and eliminates computational redundancy while achieving an adaptive balance between system throughput efficiency and resource utilization, effectively supporting efficient sampling of ultra-large-scale implicit spaces. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a flowchart of a hierarchical adaptive caching method for a large-scale sampling space provided in an embodiment of the present invention; Figure 2 This is a structural diagram of a hierarchical adaptive caching device for a large-scale sampling space provided in an embodiment of the present invention; Figure 3 This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0014] Please refer to Figure 1 This invention provides a hierarchical adaptive caching method for a large-scale sampling space, the method comprising: Step 100: Initialize the storage space and nodes of the sampling system to build a storage and operation model for data sampling and enumeration retrieval; Step 102: Perform hierarchical mapping on the sampling requests of external input according to the storage operation model, and perform real-time simplification on the sampling path nodes that have completed the hierarchical mapping to obtain the accurate positioning result of the target leaf node and a simplified node set that retains only the node navigation information. Step 104: Perform adaptive scheduling on the simplified node set to obtain a hierarchical adaptive cache set that meets resource and access requirements.

[0015] In this embodiment of the invention, the method first constructs a storage operation model by dividing a global cache pool and defining node states. Then, it uses this model to perform hierarchical and rapid retrieval of sampling requests. Simultaneously, after each layer of retrieval, an immediate thinning and release strategy is executed, erasing the parent node details immediately after the child node's computation is complete, dynamically compressing memory usage to the current sampling depth, fundamentally preventing unlimited memory expansion. Finally, based on adaptive management of access frequency, high-frequency nodes are intelligently retained while low-frequency nodes are ousted, maximizing cache hit rate within limited memory. Overall, this solution, through a collaborative mechanism of "materialization on demand, dismantling after use, and retention of hotspots," significantly reduces memory usage and eliminates computational redundancy while achieving an adaptive balance between system throughput efficiency and resource utilization, effectively supporting efficient sampling of ultra-large-scale implicit spaces.

[0016] The following description Figure 1 The execution method of each step is shown.

[0017] First, for step 100, the storage space and nodes of the sampling system are initialized to construct a storage and operation model for data sampling and enumeration retrieval.

[0018] While the basic on-demand caching scheme alleviates the initial computational pressure, it fails to recognize the topological characteristics of the sampling tree (such as differences in access frequency among different nodes), causing the cache pool to expand rapidly during the sampling process. When the sampling space becomes extremely large, even caching only accessed nodes will eventually lead to memory overflow. Therefore, this embodiment designs a storage operation model in the sampling space to complete the unified management of underlying memory resources and the initialization of basic data structures, providing stable storage support for subsequent sampling caching.

[0019] In this embodiment of the invention, the storage and operation model is constructed in the following manner: According to the preset task requirements, a contiguous physical memory block is divided into a global cache pool in the total physical memory of the sampling system, and an atomic mapping table is established in the global cache pool to represent the correspondence between logical node handles and physical memory offsets.

[0020] Specifically, upon system startup, the system first reads the total physical memory capacity of the current server. After reserving the memory space necessary for the operating system kernel, basic services, and computing engine to run, it requests a large, contiguous block of physical memory from the operating system in one go. This block serves as a dedicated global cache pool for the sampling caching system, and no further fragmented memory allocation or deallocation requests are made to the operating system throughout the process. Simultaneously, a free list for the global cache pool is constructed; this is a linked structure specifically used to record free memory blocks that have not yet been allocated.

[0021] Subsequently, an atomic mapping table is established between logical node handles and physical memory offsets: a globally unique lightweight integer handle is allocated to each possible logical node. The mapping table only stores the correspondence between the handle and the starting address of a contiguous block in the memory pool, and does not store any node business data. During memory allocation, contiguous blocks of the corresponding size are directly split from the free list of the cache pool and the mapping table is updated; during memory reclamation, the corresponding memory block is only marked as free and reinserted into the free list, without triggering system-level memory release and page reclamation operations.

[0022] To give a simple example, logical nodes are the node data in the sampling tree, and logical node handles are the unique retrieval codes corresponding to the data, such as 10086, a lightweight integer; physical memory offsets represent the precise location in memory; and atomic mapping tables are used to record the correspondence between retrieval codes and locations. When in actual use, by entering the corresponding handle, the data stored in the corresponding node can be directly queried without traversing and searching.

[0023] Furthermore, the global cache pool is divided into dual-state storage and the resulting storage space is initialized to obtain completely isolated dual-state storage areas. The dual-state storage area includes a resident weight area and a temporary detail area. The resident weight area is allocated a fixed capacity of memory to store lightweight statistical information, and the temporary detail area is allocated the remaining dynamically adjustable memory to store intermediate computational data of the nodes.

[0024] Specifically, within the global cache pool, static logical storage domains are divided. According to a configurable ratio, such as 1:9, the memory pool is divided into two completely isolated storage segments: a resident weight area and a temporary detail area. The two storage segments have independent free memory, capacity counters, and read / write locks, and do not interfere with each other.

[0025] The resident weight area uses a fixed-length array as its underlying data structure, reserving a fixed-size storage unit for each node (e.g., 12 bytes, of which 8 bytes store the node's total weight and 4 bytes store the total capacity of the corresponding subtree). After the data is written, it can only be read and not modified. The resident weight area is used to store lightweight statistical information required for node navigation. This information can reside in memory while the node is in a reduced weight state. When the node is reclaimed, the data in this area can be released.

[0026] The temporary details area adopts a hybrid data structure of "memory block pool + dynamic linked list". This area is divided into multiple fixed-size memory blocks (such as 4KB / block), and free blocks are managed by linking them together through a linked list. It supports the splicing and storage of variable-length data and the reclamation of whole blocks. It is specifically used to temporarily store intermediate quantities such as subspace partition boundaries, dimension constraints, and offsets, so as to meet the needs of high-frequency dynamic allocation and destruction.

[0027] Meanwhile, dedicated atomic read / write interfaces are implemented for each of the two storage domains to ensure data consistency in multi-threaded concurrent sampling scenarios.

[0028] Furthermore, state rules are defined and storage operations are bound to the nodes in the dual-state storage area in sequence to obtain a global node state registry that includes virtual state, full materialized state, reduced weight state and release state, as well as state transition conditions.

[0029] Specifically, based on the aforementioned dual-state storage area, the initialization state definition of the nodes is performed, and the state of all nodes is recorded in a preset global node state registry. The node states include virtual state, full materialized state, reduced weight state, and release state.

[0030] The virtual state is the initial state of all nodes. At this time, a node exists only as a concept in the logical space partitioning tree. All its data (including its own aggregate weight, the partitioning and weight of its child nodes, etc.) has not been calculated or stored in memory. A node is in this state after it is first created or completely released from memory.

[0031] The fully materialized state is the state in which a node is fully instantiated. When the node needs to be accessed, the system calculates the detailed partitioning information and weights of all its child nodes in real time. At this time, the node's "resident weight area" and "temporary detail area" are filled with data and stored completely in the memory of the pooled storage module.

[0032] Simplified weight state is an optimized state of a node. In this state, the node only retains the data in its "resident weight area" (i.e., its own aggregate weight and subtree capacity information) in memory, while intermediate data such as detailed subspace partitioning rules in its "temporary detail area" have been cleared in real time.

[0033] The release state is not strictly a "state," but rather an action and its result. It refers to the fact that all the physical memory occupied by the node (including the "resident weight area") is reclaimed to the global cache pool, the node data disappears from memory, and its logical state returns to the "virtual state."

[0034] Furthermore, this embodiment defines the following rules for the transition between node states in subsequent processing: From virtual state to full materialized state: The triggering condition is that the node cache is missing and the weight of its child nodes needs to be calculated; the execution action is to simultaneously request memory from the resident weight area and the temporary detail area to store the full data of the node.

[0035] From full materialized state to simplified weighted state: The trigger condition is that the total weight of all child nodes of the node has been calculated and written to the resident weight area of ​​the child nodes; the action is to reclaim the memory of the temporary detail area of ​​the node and retain only the data in the resident weight area.

[0036] Simplify weight state to release state: The trigger condition is that the node access popularity is lower than the threshold or the global cache capacity is insufficient; the action is to reclaim the memory of the node's resident weight area and clear the corresponding entry in the state registry.

[0037] Release state to virtual state: The trigger condition is that the node is visited again; the action is to re-register the node in the state registry and return to the initial virtual state.

[0038] In other words, the full materialized state ensures that any path can be computed and reached on demand; the simplified weighted state minimizes memory overhead while preserving navigation capabilities; and timely release ensures that the cache space can efficiently serve high-frequency hot data.

[0039] Then, for step 102, the sampling requests of external input are hierarchically mapped according to the storage operation model, and the sampling path nodes that have completed the hierarchical mapping are simplified in real time to obtain the accurate positioning result of the target leaf node and the simplified node set that retains only the node navigation information.

[0040] In this embodiment of the invention, the sampling requests from external inputs are hierarchically mapped according to the storage and operation model to obtain the precise location result of the target leaf node, including: The sampling execution context is initialized based on the core parameters contained in the sampling request, and the recursive starting point of the sampling path is determined to be the root node of the preset sampling space partitioning tree.

[0041] Specifically, the system receives sampling requests from external sources, parses and extracts core parameters such as random seed, target sampling depth, and dimensional constraints, initializes the recursive mapping context, and by default starts path exploration from the root node of the preset sampling space partitioning tree.

[0042] Furthermore, the sampling request is hierarchically mapped according to the core parameters and the node state of the current recursive node to obtain the accurate positioning result of the target leaf node.

[0043] Specifically, the mapping process includes: S21. Determine the node state of the current recursive node based on the dual-state storage area; that is, check whether the child node information of the current recursive node has been cached. S22. If the current recursive node is in a virtual state, then determine that the current recursive node is missing from the cache and perform batch calculation of child nodes and atomic writing of dual-state storage on the current recursive node to transform the current recursive node into a fully materialized parent node containing a set of virtual child nodes. After processing, proceed to step S24. Among them, the calculation of child nodes includes calculating the spatial partitioning and weight of all its child nodes. S23. If the current recursive node is in the simplified weighted state, then determine that the temporary detail area of ​​the current recursive node has been released and perform data completion and state restoration processing on the current recursive node to transform the current recursive node into a full materialized parent node containing a set of virtual state child nodes. After processing, execute step S24. S24. If the current recursive node is in a fully materialized state, then it is determined that the data in the permanent weight area and the temporary detail area of ​​the current recursive node are complete. Based on the core parameters, a binary search is performed in the permanent weight area of ​​the current recursive node to obtain the target child node corresponding to the core parameters, and step S25 is executed. S25. Determine whether the target child node is a leaf node. If so, determine the target child node as the target leaf node and output the data corresponding to the target leaf node. Otherwise, determine the target child node as the current recursive node and return to step S21.

[0044] For example, suppose that after processing S22 and S23, the current recursive node has been transformed into a fully materialized state, and the node's resident weight region includes three child nodes with weights of 20, 30, and 50 respectively. The random seed in the sampling request is 35. Then, the weights of the three child nodes are summed to obtain 20, 50, and 100. A binary search is performed using the random seed. Since 35 > 20 and 35 < 50, it can be determined that the target leaf node falls within the second child node. It is then determined whether the second child node is a leaf node that cannot be further divided or has reached the target search depth. If so, the corresponding sampling result is output. Otherwise, the above S21-S25 operations are repeated for the second child node until the final target leaf node is found.

[0045] While performing the above process, each time the current recursive node completes a round of processing and enters the next node, the sampling path node that has completed the hierarchical mapping is immediately simplified.

[0046] In this embodiment of the invention, the simplification process includes: generating simplification instructions for path nodes that have completed data writing and child node weight calculation based on the writing progress of the path nodes that are currently undergoing hierarchical mapping; According to the simplification instruction, all data of the path node stored in the temporary details area is cleared, and the storage space after data clearing is returned to the global cache pool, resulting in a simplified node that only stores path navigation information in the resident weight area.

[0047] As described in the definition of node states above, once a node in the "fully materialized state" has completed the calculation and materialization of all its child nodes, the system will immediately perform instant simplification and release, reverting its state to the simplified weighted state, and simultaneously updating the global node state registry. This state only retains the data in its "resident weight area," that is, it retains the key information necessary for navigation, but memory usage is significantly reduced.

[0048] In summary, the above-described cyclical process of positioning, simplification, and downward exploration will continue until the leaf node is finally reached, completing the entire mapping from the random seed to the specific element. At the same time, a simplified set of nodes that retains only navigation information is obtained, which can be used to improve processing efficiency when sampling and mapping the node again in the future.

[0049] For step 104, adaptive scheduling processing is performed on the simplified node set to obtain a hierarchical adaptive cache set that meets resource and access requirements.

[0050] In this embodiment of the invention, the adaptive scheduling process includes: performing statistical analysis on the execution context of the nodes in the streamlined node set to obtain the node's reference count, access popularity, and recycling flag; calculating the node's access popularity threshold based on the current available memory in the global cache pool; retaining all data of the recyclable node in the resident weight area when the access popularity of the recyclable node is higher than the threshold; and clearing all data of the recyclable node in the resident weight area and returning the corresponding memory to the global cache pool to convert the node into a released state when the access popularity of the recyclable node is lower than the threshold.

[0051] Specifically, during system operation, the number of visits to each node is counted in real time. Combined with the node's level information in the partitioning tree, the access frequency popularity index of the node is calculated and predicted. Nodes with higher levels are assigned higher basic popularity weights by default.

[0052] Meanwhile, the system will adaptively adjust the hot threshold of cached nodes based on the current available physical memory capacity. When the system has sufficient memory (low utilization), the module will lower this threshold, which means that the system is willing to keep more nodes in the cache in order to pursue higher performance. When the system has tight memory (high utilization or close to the limit), the module will raise this threshold, which means that the system will adopt a stricter retention standard, only keeping the most frequently accessed nodes and actively releasing more memory space to prevent overflow.

[0053] Furthermore, for high-frequency access nodes with a popularity level above the threshold, their release operation is exempted, allowing them to remain in memory in a simplified weighted state for a long time, eliminating computational redundancy from subsequent repeated accesses; for low-frequency access nodes with a popularity level below the threshold, a "use and dismantle" strategy is implemented, that is, after they complete the current sampling path navigation, their physical memory in the resident weight area is immediately reclaimed, and the node state is completely reset to the initial release state.

[0054] Meanwhile, when the total cache pool usage reaches the preset limit, the resident streamlined weighted nodes are eliminated in order of increasing popularity until the memory usage drops back to a safe range, ensuring that the system will not experience memory overflow due to cache expansion.

[0055] In summary, by establishing a node-based hierarchical management and adjustment mechanism, the system can adjust the cache size according to different memory limitations, making full use of limited storage resources, thereby supporting efficient computing under different hardware constraints.

[0056] Please refer to Figure 2 This invention provides a hierarchical adaptive caching device for a large-scale sampling space, the device comprising: Initialization module 200 is used to initialize the storage space and nodes of the sampling system and build a storage and operation model for data sampling and enumeration retrieval. The sampling module 202 is used to perform hierarchical mapping on the sampling requests of external input according to the storage operation model, and to perform real-time simplification on the sampling path nodes that have completed the hierarchical mapping, so as to obtain the accurate positioning result of the target leaf node and the simplified node set that retains only the node navigation information. The scheduling module 204 is used to perform adaptive scheduling processing on the simplified node set to obtain a hierarchical adaptive cache set that meets resource and access requirements.

[0057] In this embodiment of the invention, the initialization process of the storage space and nodes of the sampling system to construct a storage and operation model for data sampling and enumeration retrieval includes: According to the preset task requirements, the total physical memory of the sampling system is divided into contiguous physical memory blocks as a global cache pool, and an atomic mapping table is established in the global cache pool to represent the correspondence between logical node handles and physical memory offsets. The global cache pool is divided into two-state storage and the resulting storage space is initialized to obtain completely isolated two-state storage areas. The two-state storage areas include a resident weight area and a temporary detail area. The resident weight area is allocated a fixed amount of memory to store lightweight statistical information, and the temporary detail area is allocated the remaining dynamically adjustable memory to store intermediate computational data of the nodes. The nodes in the dual-state storage area are sequentially defined with state rules and bound to storage operations to obtain a global node state registry that includes virtual state, full materialized state, reduced weight state, and release state, as well as state transition conditions.

[0058] In this embodiment of the invention, the sampling requests from external inputs are hierarchically mapped according to the storage and operation model to obtain the precise location result of the target leaf node, including: The sampling execution context is initialized based on the core parameters contained in the sampling request, and the recursive starting point of the sampling path is determined to be the root node of the preset sampling space partitioning tree. Based on the core parameters and the node state of the current recursive node, the sampling request is mapped hierarchically to obtain the accurate location result of the target leaf node.

[0059] In this embodiment of the invention, the step of performing hierarchical mapping of the sampling request based on the core parameters and the node state of the current recursive node to obtain the precise location result of the target leaf node includes: S21. Determine the node state of the current recursive node based on the dual-state storage area; S22. If the current recursive node is in virtual state, determine that the current recursive node cache is missing and perform batch calculation of child nodes and atomic writing of dual-state storage on the current recursive node to transform the current recursive node into a full materialized parent node containing a set of virtual state child nodes. After processing, execute step S24. S23. If the current recursive node is in the simplified weighted state, then determine that the temporary detail area of ​​the current recursive node has been released and perform data completion and state restoration processing on the current recursive node to transform the current recursive node into a full materialized parent node containing a set of virtual state child nodes. After processing, execute step S24. S24. If the current recursive node is in a fully materialized state, then it is determined that the data in the permanent weight area and the temporary detail area of ​​the current recursive node are complete. Based on the core parameters, a binary search is performed in the permanent weight area of ​​the current recursive node to obtain the target child node corresponding to the core parameters, and step S25 is executed. S25. Determine whether the target child node is a leaf node. If so, determine the target child node as the target leaf node and output the data corresponding to the target leaf node. Otherwise, determine the target child node as the current recursive node and return to step S21.

[0060] In this embodiment of the invention, the real-time simplification of the sampling path nodes that have completed the hierarchical mapping includes: Based on the writing progress of the path nodes that are currently undergoing hierarchical mapping, generate simplified instructions for the path nodes that have completed data writing and child node weight calculation. According to the simplification instruction, all data of the path node stored in the temporary details area is cleared, and the storage space after data clearing is returned to the global cache pool, resulting in a simplified node that only stores path navigation information in the resident weight area.

[0061] In this embodiment of the invention, the adaptive scheduling process performed on the streamlined node set to obtain a hierarchical adaptive cache set that meets resource and access requirements includes: The execution context of the nodes in the simplified node set is statistically analyzed to obtain the node's reference count, access popularity, and recycling flag. The access popularity threshold of a node is calculated based on the total available memory in the global cache pool. When the access popularity of a recyclable node exceeds the threshold, all data of the recyclable node in the resident weight area is retained. When the access popularity of a recyclable node is lower than the threshold, all data of the recyclable node in the resident weight area is cleared, and the corresponding memory is returned to the global cache pool so that the node is converted to a released state.

[0062] It should be noted that the hierarchical adaptive caching device for large-scale sampling space provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the hierarchical adaptive caching device for large-scale sampling space provided in the above embodiments and the hierarchical adaptive caching method embodiments for large-scale sampling space belong to the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0063] Embodiments of this application also provide a computer device, please refer to... Figure 3 The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the hierarchical adaptive caching method for large-scale sampling space provided in the above-described method embodiments.

[0064] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the hierarchical adaptive caching method for large-scale sampling space provided in the above-described method embodiments.

[0065] Embodiments of this application also provide a computer program product comprising a computer program, wherein a processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program such that the computer device performs any of the hierarchical adaptive caching methods for large-scale sampling spaces described in the above embodiments.

[0066] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0067] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0068] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A hierarchical adaptive caching method for large-scale sampling spaces, characterized in that, The method includes: The storage space and nodes of the sampling system are initialized to construct a storage and operation model for data sampling and enumeration retrieval. According to the storage and operation model, the sampling requests of external inputs are mapped in layers, and the sampling path nodes that have completed the layer mapping are simplified in real time to obtain the accurate positioning result of the target leaf node and the simplified node set that retains only the node navigation information. The simplified node set is subjected to adaptive scheduling to obtain a hierarchical adaptive cache set that meets resource and access requirements.

2. The method as described in claim 1, characterized in that, The initialization process for the storage space and nodes of the sampling system, and the construction of a storage and operation model for data sampling and enumeration retrieval, includes: According to the preset task requirements, the total physical memory of the sampling system is divided into contiguous physical memory blocks as a global cache pool, and an atomic mapping table is established in the global cache pool to represent the correspondence between logical node handles and physical memory offsets. The global cache pool is divided into two-state storage and the resulting storage space is initialized to obtain completely isolated two-state storage areas. The two-state storage areas include a resident weight area and a temporary detail area. The resident weight area is allocated a fixed amount of memory to store lightweight statistical information, and the temporary detail area is allocated the remaining dynamically adjustable memory to store intermediate computational data of the nodes. The nodes in the dual-state storage area are sequentially defined with state rules and bound to storage operations to obtain a global node state registry that includes virtual state, full materialized state, reduced weight state, and release state, as well as state transition conditions.

3. The method as described in claim 2, characterized in that, Based on the storage operation model, the sampling requests from external inputs are mapped hierarchically to obtain the precise location result of the target leaf node, including: The sampling execution context is initialized based on the core parameters contained in the sampling request, and the recursive starting point of the sampling path is determined to be the root node of the preset sampling space partitioning tree. Based on the core parameters and the node state of the current recursive node, the sampling request is mapped hierarchically to obtain the accurate location result of the target leaf node.

4. The method as described in claim 3, characterized in that, The step of performing hierarchical mapping of sampling requests based on the core parameters and the node state of the current recursive node to obtain the precise location result of the target leaf node includes: S21. Determine the node state of the current recursive node based on the dual-state storage area; S22. If the current recursive node is in virtual state, determine that the current recursive node cache is missing and perform batch calculation of child nodes and atomic writing of dual-state storage on the current recursive node to transform the current recursive node into a full materialized parent node containing a set of virtual state child nodes. After processing, execute step S24. S23. If the current recursive node is in the simplified weighted state, then determine that the temporary detail area of ​​the current recursive node has been released and perform data completion and state restoration processing on the current recursive node to transform the current recursive node into a full materialized parent node containing a set of virtual state child nodes. After processing, execute step S24. S24. If the current recursive node is in a fully materialized state, then it is determined that the data in the permanent weight area and the temporary detail area of ​​the current recursive node are complete. Based on the core parameters, a binary search is performed in the permanent weight area of ​​the current recursive node to obtain the target child node corresponding to the core parameters, and step S25 is executed. S25. Determine whether the target child node is a leaf node. If so, determine the target child node as the target leaf node and output the data corresponding to the target leaf node. Otherwise, determine the target child node as the current recursive node and return to step S21.

5. The method as described in claim 3, characterized in that, The instantaneous simplification of the sampling path nodes after completing the hierarchical mapping includes: Based on the writing progress of the path nodes that are currently undergoing hierarchical mapping, generate simplified instructions for the path nodes that have completed data writing and child node weight calculation. According to the simplification instruction, all data of the path node stored in the temporary details area is cleared, and the storage space after data clearing is returned to the global cache pool, resulting in a simplified node that only stores path navigation information in the resident weight area.

6. The method as described in claim 2, characterized in that, The adaptive scheduling process performed on the streamlined node set yields a hierarchical adaptive cache set that satisfies resource and access requirements, including: The execution context of the nodes in the simplified node set is statistically analyzed to obtain the node's reference count, access popularity, and recycling flag. The access popularity threshold of a node is calculated based on the total available memory in the global cache pool. When the access popularity of a recyclable node exceeds the threshold, all data of the recyclable node in the resident weight area is retained. When the access popularity of a recyclable node falls below the threshold, all data of the recyclable node in the resident weight area is cleared, and the corresponding memory is returned to the global cache pool, so that the node is converted to a released state.

7. A hierarchical adaptive caching device for a large-scale sampling space, characterized in that, The device includes: The initialization module is used to initialize the storage space and nodes of the sampling system and build a storage and operation model for data sampling and enumeration retrieval. The sampling module is used to perform hierarchical mapping of sampling requests from external inputs according to the storage and operation model, and to perform real-time simplification of the sampling path nodes that have completed the hierarchical mapping, so as to obtain the accurate positioning result of the target leaf node and a simplified node set that retains only the node navigation information. The scheduling module is used to perform adaptive scheduling processing on the simplified node set to obtain a hierarchical adaptive cache set that meets resource and access requirements.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images