A data mining method and device for intelligent search of mass information

Abstract

The application discloses a data mining method and equipment for intelligent search of mass information. The method comprises the following steps: analyzing a target query instruction to generate an abstract syntax tree and extracting time and frequency characteristics; calculating an access probability and asynchronously loading a target data block when a trigger threshold is reached; obtaining a target data block identifier preloaded to a memory buffer pool to form a resident state bitmap; intercepting and rewriting an initial execution plan, splitting a single scanning operator into a scanning branch for the memory buffer pool and a scanning branch for unloaded disk data, and issuing execution; and statistically counting an actual memory hit rate and dynamically adjusting a trigger threshold based on a deviation value in a closed loop. The application can reduce cross-network input / output requests and physical seek times, reduce disk congestion probability, suppress first-round query delay peak, and improve overall throughput of a distributed database system.

Description

Technical Field

[0001] This application relates to the field of distributed database query technology, and more specifically, to a data mining method and device for intelligent search of massive amounts of information. Background Technology

[0002] In industrial IoT and large-scale government data processing systems, the underlying database stores massive amounts of device status logs and historical information. When upper-layer applications initiate complex, multi-dimensional query requests with high concurrency, the query conditions often span different time windows.

[0003] Currently, most mainstream relational or distributed databases employ cost-based or rule-based query optimizers to generate query execution plans. These optimizers typically rely on periodically collected static statistics for storage cost estimation. However, due to high concurrency and frequent shifts in query hotspots, static statistics cannot reflect the real-time physical boundaries between hot and cold data. Existing query processing mechanisms frequently generate suboptimal execution plans with full table scans or large-scale cross-node index scans when handling complex queries, leading to disk I / O congestion. Traditional passive caching mechanisms can only load data into memory after it has been read, failing to handle latency spikes during the initial query round, resulting in overall system lag. Summary of the Invention

[0004] This application provides a data mining method and device for intelligent search of massive information, which solves the problem that existing distributed database optimizers cannot perceive the real-time access time sequence characteristics of data, resulting in suboptimal execution plans and causing a large amount of disk I / O congestion.

[0005] This application provides a data mining method for intelligent search of massive information, comprising the following steps: Parsing a target query instruction to generate an abstract syntax tree (AST), and extracting the time dimension field and query frequency information from the predicate conditions of the AST. Based on the time dimension field and the query frequency information, calculating the access probability of a target data block; when the access probability reaches a set trigger threshold, asynchronously loading the target data block into a memory buffer pool. Obtaining the data block identifier of the target data block preloaded into the memory buffer pool to form a resident state bitmap. Intercepting the initial execution plan generated for the target query instruction, rewriting the initial execution plan according to the resident state bitmap, splitting a single scan operator into a first scan branch facing the memory buffer pool and a second scan branch facing unloaded disk data, and then sending the rewritten execution plan for execution.

[0006] Furthermore, the step of parsing the target query instruction to generate an abstract syntax tree and extracting the time dimension field and query frequency information from the predicate conditions of the abstract syntax tree includes: performing lexical and syntactic analysis on the target query instruction to construct the abstract syntax tree corresponding to the target query instruction; traversing the filter nodes of the abstract syntax tree to identify the time dimension field containing the time range attribute; and counting the number of times the target query instruction with the same hash signature appears within a preset time window to determine the query frequency information.

[0007] Further, the step of calculating the access probability of the target data block based on the time dimension field and the query frequency information includes: obtaining the baseline popularity value corresponding to the target data block, and the time difference between the current query time point and the last access time point indicated by the time dimension field; obtaining the time decay factor corresponding to the current system load, wherein the time decay factor characterizes the decay rate of the baseline popularity value over time; using the product of the time difference value and the time decay factor as a decay penalty term, and subtracting the decay penalty term from the baseline popularity value to obtain a first intermediate probability value; using the product of the query frequency information and the preset frequency weight as a frequency gain term, and adding the first intermediate probability value to the frequency gain term to obtain the access probability of the target data block.

[0008] Further, obtaining the data block identifier of the target data block preloaded into the memory buffer pool to form a resident state bitmap includes: allocating contiguous memory space in the memory buffer pool as a global bitmap region; in response to a notification that the asynchronous loading of the target data block is complete, extracting the physical page number corresponding to the data block identifier; mapping the physical page number to a specific bit within the global bitmap region using a hash mapping function, and setting the specific bit to a valid state to generate the resident state bitmap.

[0009] Further, the step of rewriting the initial execution plan according to the resident state bitmap, splitting a single scan operator into a first scan branch facing the memory buffer pool and a second scan branch facing unloaded disk data, and then sending the rewritten execution plan for execution, includes: extracting the block index set of the physical blocks to be scanned in the single scan operator; performing a bitwise AND operation between the block index set and the resident state bitmap to separate a first subset of block indexes in the resident state and a second subset of block indexes in the non-resident state; generating a corresponding hash addressing operator for the first subset of block indexes as the first scan branch, and generating a corresponding physical disk cursor read operator for the second subset of block indexes as the second scan branch; inserting a joint reorganization operator at the top level of the first scan branch and the second scan branch to merge the data streams and form the rewritten execution plan for execution.

[0010] Furthermore, the data mining method further includes: periodically calculating the actual memory hit rate of the query task in the memory buffer pool within a preset time period; calculating the deviation between the actual memory hit rate and the expected hit rate; and dynamically adjusting the trigger threshold used when calculating the access probability based on the deviation value in a closed loop.

[0011] Furthermore, the trigger threshold used in calculating the access probability based on the dynamic closed-loop adjustment of the deviation value includes: when the actual memory hit rate is less than a first preset hit rate threshold and the buffer pool memory balance is greater than a preset capacity threshold, it is determined that the system is in an underloaded state, and the trigger threshold is lowered according to a preset step size ratio to increase the number of asynchronously loaded data blocks; when the actual memory hit rate is greater than a second preset hit rate threshold and the buffer pool memory balance is less than the preset capacity threshold, it is determined that the system is in an overload jitter state, and the trigger threshold is raised according to the increasing multiple of the exponential backoff algorithm to suppress the loading of invalid data blocks; wherein, the first preset hit rate threshold is less than the second preset hit rate threshold.

[0012] This application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the above-described data mining method for intelligent search of massive information.

[0013] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned data mining method for intelligent search of massive amounts of information.

[0014] This application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned data mining method for intelligent search of massive amounts of information.

[0015] This application intercepts the target query command and parses it to generate an abstract syntax tree, extracting temporal predicate features to accurately calculate the data access probability. This allows the system to proactively send asynchronous preloading instructions for data blocks to the underlying nodes before query execution. Based on the feedback memory resident status bitmap, the system physically rewrites the initial execution plan, splitting a single scan into a multi-way joint computation branch consisting of a memory hit scan and a disk supplement scan. This mechanism reduces cross-network disk I / O requests and physical seek times in high-concurrency scenarios, eliminating computational blocking caused by page faults at the execution level. By introducing an adjustment model based on the closed-loop deviation of memory hit rate, the threshold of the preloading model can be corrected under load fluctuations, suppressing invalid memory resource occupation and reducing the response latency of massive time-series data mining and searching. Attached Figure Description

[0016] Figure 1 This is a flowchart of the data mining method provided in an embodiment of the present invention.

[0017] Figure 2 This is a logical structure block diagram of the data mining system provided in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0019] In the diagram: 201 is the query parsing and feature extraction module, 202 is the predictive preloading controller, 203 is the adaptive rewriting engine, 204 is the feedback regulator, 301 is the memory, 302 is the processor, and 303 is the communication bus. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The data mining and query execution method and device provided in this application are mainly deployed in a distributed cluster architecture. At the physical architecture level, the entire cluster includes a coordinating node responsible for overall scheduling and distributed data nodes responsible for actual underlying data storage and operator execution. The coordinating node and each data node establish communication links through a cluster network communication module and a data synchronization bus. The underlying nodes are equipped with physical storage media, and a memory buffer pool based on a least recently used eviction algorithm is maintained on top of them to provide buffer support for the computation of massive amounts of data.

[0022] like Figure 1 As shown in the figure, this application provides a data mining method for intelligent search of massive information, which includes the following steps.

[0023] Step S101: Parse the target query instruction to generate an abstract syntax tree, and extract the time dimension field and query frequency information from the predicate conditions of the abstract syntax tree.

[0024] Upon receiving a structured query command, the system's query parsing and feature extraction module 201 in the coordinating node performs lexical scanning and syntactic analysis, converting the command into an abstract syntax tree with hierarchical logical relationships. In this tree-like data structure, leaf nodes represent specific column fields or constant values, while non-leaf nodes represent logical or comparison operators. The system traverses the filtering nodes in the abstract syntax tree, specifically searching for time attribute nodes in predicate conditional clauses. By scanning relational operators such as greater than or equal to, or less than or equal to, it identifies a set of time dimension fields containing time range attributes. These time dimension fields encompass the time filtering range required by the target query.

[0025] While extracting time-dimensional features, the system maintains a global query fingerprint record table locally. A unique hash signature is generated for the target query instruction template after removing specific constant values ​​using hash operations. The system backtracks the historical occurrences of target query instructions with the same hash signature within a preset time window, counting the number of times they are triggered to determine query frequency information. Through in-depth analysis of the syntax tree structure and separation of structure and constants, a stable behavioral pattern foundation can be abstracted for the ever-changing time-series analysis queries, thus providing a data basis for subsequent popularity prediction.

[0026] Step S102: Based on the time dimension field and the query frequency information, calculate the access probability of the target data block. When the access probability reaches the set trigger threshold, asynchronously load the target data block into the memory buffer pool.

[0027] After acquiring the predictive features across various dimensions, the system initiates predictive preloading controller 202 to perform predictive calculations on the target data blocks at the corresponding underlying physical layer. The system obtains the baseline heat value corresponding to the target data block, which represents the inherent weight of the data formed during historical usage. The system extracts the time difference between the current query time and the last access time of the data block, as indicated in the time dimension field. The system reads the system load metric of the current physical node to determine the corresponding time decay factor. This time decay factor represents the physical rate at which the baseline heat value naturally decays over time. The system then performs a comprehensive deduction using its internal probability prediction logic.

[0028] The system performs steps such as combining the difference between the current time and the last access time with a time decay factor to perform negative exponential calculations to establish a prediction model. The system uses the following formula to calculate the access probability:

[0029] ;

[0030] In this formula, P hit C represents the calculated access probability of the target data block; req This indicates the actual query frequency information of the current field set extracted in the previous steps within the most recent monitoring period; T represents the historical peak frequency of this feature dimension; now Indicates the current system timestamp generated during extraction; T last This indicates the timestamp of the last time the target data block was queried; This represents the time decay factor controlled by system load feedback. and These represent the basic weight coefficients set by the system.

[0031] In this logical structure, P hitAs a probability representation between 0 and 1, it is composed of two gain terms. The difference between the current time and the last access time, combined with a time decay factor and subjected to negative exponential operation, constitutes the first intermediate probability value with a time penalty attribute. The ratio of the actual query frequency to its historical peak value, combined with a preset frequency weight, is weighted to form a frequency gain term representing the increase in popularity. By adding the first intermediate probability value and the frequency gain term, a smooth target access probability with timely responsiveness is obtained. In the embodiments of this application, the parameter... The preferred value range for the parameter is 0.3 to 0.5. The preferred value range is 0.5 to 0.7, and the sum of the two is 1, ensuring that the prediction model has normalization characteristics in both mathematical and physical senses.

[0032] When the calculated access probability P hit When the threshold value (Th) exceeds the system's set trigger threshold, the system converts the primary key range of the target data block located on the disk storage into the corresponding physical block address of the media. The system encapsulates this physical block address into a pre-read instruction and pushes it into the pre-read task queue of the data node's memory buffer pool. Based on this queue, the underlying media driver uses an independent asynchronous I / O thread to load the binary data stream of the data block into pre-allocated memory buffer pool page frames. Through this dual-track dynamic decay prediction and asynchronous execution chain, data transmission latency can be eliminated before the backbone query operation reaches the data node, mitigating the seek storm caused by instantaneous high concurrency on the disk head.

[0033] Step S103: Obtain the data block identifier of the target data block that has been preloaded into the memory buffer pool to form a resident state bitmap.

[0034] After the asynchronous loading task is submitted, the system needs to provide precise physical boundary awareness for subsequent optimizer rewriting. In response to the hardware interrupt or software callback notification issued by the underlying asynchronous driver indicating that the data block loading is complete, the system obtains the data block identifier corresponding to the target data block and extracts its physical page number residing in the current memory.

[0035] Within the memory buffer pool of the data node, the system pre-allocates contiguous physical memory space as a bitmap region representing the global loading state. The system calls a preset hash mapping function to map the extracted physical page number to a specific bit offset within the global bitmap region. Using memory-level atomic set operations, this specific bit is set to a high-level active state. As the loaded data increases, this contiguous memory region gradually forms a complete resident state bitmap.

[0036] By maintaining a lightweight resident state bitmap in memory, the system can respond to hit detection of massive data blocks. The lightweight state tracking structure, independent of the main business execution flow, avoids performing traversal comparisons during the query process, providing a benchmark for subsequent operator segmentation.

[0037] Step S104: Intercept the initial execution plan generated for the target query instruction, rewrite the initial execution plan according to the resident state bitmap, split the single scan operator into a first scan branch facing the memory buffer pool and a second scan branch facing unloaded disk data, and send the rewritten execution plan for execution.

[0038] The initial execution plan generated by existing optimizers typically uses a single scan operator to fetch data from the relevant partitions. In this step, the plan adaptive rewriting engine 203 inside the coordinating node is pre-activated, intercepting the initial execution plan at the optimizer output backend and modifying the operator graph structure in the plan tree.

[0039] The adaptive rewrite engine 203 parses the initial execution plan and extracts the set of block indices of the physical blocks to be scanned specified in a single scan operator. The engine then queries and calls the global resident state bitmap generated in the preceding steps. It performs a low-level bitwise AND operation with the obtained block index set and the resident state bitmap. Using the result of the Boolean operation, the engine separates the originally contiguous queue of blocks to be read into a first subset of indices in a resident state and a second subset of indices still in a non-resident state.

[0040] If the first index subset is detected to be non-empty, it indicates that the target data block is partially ready due to the warm-up mechanism. The system adaptively rewrites the startup graph structure modification logic of engine 203, removing the single scan operator from the initial execution plan. For the first index subset, the system generates a corresponding hash addressing operator in the syntax tree as the first scan branch, responsible for directly connecting to the memory buffer pool. For the second index subset, the system generates a corresponding physical disk cursor read operator as the second scan branch, and adds boundary predicate conditions to it, ensuring that it only performs the fetch operation within the physical disk file cursor range that has not been preloaded.

[0041] After establishing the two branches, the adaptive rewrite engine 203 inserts a multi-way joint reassembly operator at the top level of the execution flow of the first and second scan branches. This joint reassembly operator is responsible for aggregating the data streams output by the two branches. Specifically, it asynchronously acquires data from high-speed memory streams and low-speed disk streams through a non-blocking data pipeline mechanism, and performs merging and consistency checks based on the original primary key set of the target data block, thereby seamlessly splicing together a complete data stream output. The adaptive rewrite engine 203 serializes the new execution plan containing the multi-way joint operator and distributes it to the distributed data nodes for execution.

[0042] In real-world physical execution environments, traditional physical disk I / O random reads take approximately 5 to 10 milliseconds, and are prone to queuing and congestion under high concurrency. This embodiment addresses this by performing operator splitting and replacement within the engine, allowing most data with a high hit probability to be directly addressed via memory-level pointers through the first scan branch. Only the remaining cold data is pushed to the second scan branch. This mechanism eliminates long page faults caused by a single operator encountering a cold block, breaking the bottleneck of single-access media access.

[0043] Step S105: Periodically calculate the actual memory hit rate of the query task in the memory buffer pool within a preset time period, and dynamically adjust the trigger threshold used when calculating the access probability in a closed loop.

[0044] The system not only provides forward command pushdown but also includes a feedback regulator 204 for reverse correction. The feedback regulator 204 has a built-in counter module that periodically samples the execution nodes according to a set feedback time window. For example, the feedback time window can be set to trigger every 5 minutes.

[0045] At the end of each statistical period, the feedback regulator 204 retrieves the buffer pool read / write logs. The system performs a division operation, dividing the number of data blocks actually queried and consumed by the upper layer after preloading by the total number of preloading command triggers issued by the predictive preloading controller 202. The actual memory hit rate within that time window is calculated using the following formula:

[0046] ;

[0047] Where R represents the actual memory hit rate obtained from statistics within the current monitoring period; N hit N represents the number of data blocks actually queried and consumed by the upper layer after the system executes the preload instruction. load This indicates the total number of preload command triggers issued by the predictive preload controller 202.

[0048] After calculating this value, the feedback regulator 204 makes a joint judgment based on the system's underlying resource level. The system performs dynamic closed-loop adjustment by comparing the deviation between the actual memory hit rate and the preset hit rate target.

[0049] When the system detects that the actual memory hit rate is less than the first preset hit rate threshold, and simultaneously obtains that the remaining memory in the underlying buffer pool is greater than the preset capacity threshold set by the system, the feedback regulator 204 determines that the current prediction model is in an underloaded state. The system issues a downlink control command to lower the trigger threshold Th in the model according to a preset step size ratio. Lowering this threshold makes the probability formula easier to satisfy, thereby expanding the size of the asynchronously loaded data block and making full use of idle memory to provide cache hit redundancy.

[0050] Conversely, when the feedback regulator 204 calculates that the actual memory hit rate R is consistently lower than the lower limit of the second preset hit rate threshold, and the current buffer pool memory balance is lower than the preset capacity threshold, the system determines that the current prediction model has fallen into an overload jitter state. In a preferred embodiment, the lower limit for determining the actual memory hit rate R is set to 0.6. If the actual memory hit rate R collected in three consecutive feedback time windows is lower than 0.6, the feedback regulator 204 will determine that the preheating logic has an aggressive loading risk and then activate the suppression logic. The system increases the trigger threshold Th according to the increment multiplier generated by the exponential backoff algorithm. For example, the trigger threshold Th is increased in a stepwise manner with a dynamic increment of 15%. The system simultaneously increases the time decay factor by a multiple. The value of .

[0051] It should be noted that the first preset hit rate threshold must be lower than the second preset hit rate threshold to allow for a physical hysteresis range in the control system to prevent frequent oscillations. By constructing this dual closed-loop adjustment link based on resource awareness and effectiveness bias, the system can reduce the reliance of the probabilistic model on early historical hotspots, eliminate invalid memory requests, and avoid excessive swapping and thrashing of memory pages during massive high-frequency access.

[0052] Based on the above embodiments, such as Figure 2 As shown, this application also provides a data mining system. The system's logical architecture includes multiple functionally coordinated modular components. The system mainly comprises a query parsing and feature extraction module 201, a predictive preloading controller 202, a plan adaptive rewriting engine 203, and a feedback regulator 204.

[0053] The query parsing and feature extraction module 201, acting as a request receiving front-end unit, is configured to perform lexical decomposition on the incoming raw instructions, generate a structured tree-like node model, and identify and extract the relevant fields and time interval parameters. The predictive preloading controller 202 logically establishes a data connection with the query parsing and feature extraction module 201, and internally integrates a computing power evaluation chip and multi-dimensional decay algorithm logic. This predictive preloading controller 202 receives parameters from the front end, performs heat calculations for the underlying blocks, and controls the drive hardware to perform pre-emptive cross-layer data transfer tasks. The adaptive rewriting engine 203, as the core hub of backend optimization processing, is deployed in the physical operator generation and distribution chain. It obtains precise residency status by probing the bitmap region fed back by the memory array, and completes the reorganization and distribution of the plan tree by cutting and replacing logical operator branches. The feedback regulator 204 is connected between the data execution side and the parameter feedback loop of the predictive preloading controller 202. It calculates statistical indicators by periodically collecting hit point logs, generates gain or suppression correction instructions, and injects them back into the predictive model.

[0054] With the coordinated operation of the various physical modules of the system, the underlying physical memory data is retrieved and separated in advance at a high frequency. Due to the intervention of operator-level replacement operations, a seamless flow framework for query conditions across multiple storage media is constructed, which expands the system's retrieval and concurrent processing capabilities when dealing with time-series data.

[0055] like Figure 3 As shown, another embodiment of this application provides an electronic device for deploying the aforementioned data mining system. This electronic device, serving as the physical carrier in a distributed cluster environment, typically manifests as a rack server or a high-configuration coordinating computer.

[0056] The electronic device mainly includes an interconnected memory 301, a processor 302, and an internal communication bus 303 at the physical level.

[0057] The processor 302, as the core computing entity of the device, can employ a central processing unit, application-specific integrated circuits (ASICs), and other computing chips capable of supporting large-scale concurrent control instructions. This processor is connected to the communication bus 303 via pins, receiving and processing interrupts and digital signals sent by various peripheral circuits and network cards. The memory 301 includes volatile RAM and non-volatile disk or flash memory. The volatile RAM is primarily allocated as a buffer pool for storing massive amounts of preheating data and storing the global resident state bitmap, as described in the aforementioned steps.

[0058] In actual operation, the memory 301 stores computer program instructions with specific logical steps. When the processor 302 receives a power-on startup or instruction call request, it reads and executes the computer program via the bus. During its execution, the physical hardware circuitry, in conjunction with the underlying driver environment, can completely reproduce the step control of all data mining and execution plan rewriting methods disclosed in the aforementioned embodiments. Through the physical cooperation between the high-performance processor and the memory pool, the complex warm-up calculations in the system software architecture are effectively implemented.

[0059] Based on the same logic and implementation environment described above, this application also provides a computer-readable storage medium. This storage medium is a non-transient solid-state storage device, such as a read-only memory, random access memory, magnetic hard disk drive, or optical reading device. Computer program code modules are embedded and recorded on this computer-readable storage medium. When it is docked or mounted to a terminal system with a processor hardware environment, and the computer program is scheduled for execution, it can guide the terminal system's computing unit and storage controller to complete the various processing actions of the data mining method for intelligent search of massive information described in the above embodiments.

[0060] Furthermore, this application also provides a computer program product. This product consists of a software package including microinstructions and code segments. This software package can be physically packaged in an optical disc or removable storage medium, or it can be packaged and stored on a cloud storage distribution node on a network via a predetermined transmission protocol. When the processor of an external device loads this computer program product and enters the compilation, interpretation, and execution phase, it can also drive the underlying hardware device to fully execute all the method steps based on operator rewriting and probability prediction disclosed in the embodiments of this application. This program product enables the data mining optimization mechanism of this application to be flexibly deployed in heterogeneous cloud-native platforms or edge clusters, broadening the general implementation prospects in the field of industrial time-series big data.

Claims

1. A data mining method for intelligent search of massive amounts of information, characterized in that, include: Parse the target query instruction to generate an abstract syntax tree, and extract the time dimension field and query frequency information from the predicate conditions of the abstract syntax tree; Based on the time dimension field and the query frequency information, the access probability of the target data block is calculated. When the access probability reaches the set trigger threshold, the target data block is asynchronously loaded into the memory buffer pool. Obtain the data block identifier of the target data block that has been preloaded into the memory buffer pool to form a resident state bitmap; The initial execution plan generated for the target query instruction is intercepted. The initial execution plan is rewritten according to the resident state bitmap. The single scan operator is split into a first scan branch facing the memory buffer pool and a second scan branch facing unloaded disk data. The rewritten execution plan is then sent out for execution.

2. The method as described in claim 1, characterized in that, The parsing of the target query instruction generates an abstract syntax tree, and the extraction of the time dimension field and query frequency information from the predicate conditions of the abstract syntax tree includes: Lexical and syntactic analysis is performed on the target query instruction to construct the abstract syntax tree corresponding to the target query instruction; Traverse the filter nodes of the abstract syntax tree to identify the time dimension field that contains the time range attribute; The frequency of the query is determined by counting the number of times the target query instruction with the same hash signature appears within a preset time window.

3. The method as described in claim 1, characterized in that, The step of calculating the access probability of the target data block based on the time dimension field and the query frequency information includes: Obtain the time difference between the current query time and the last access time indicated by the time dimension field, as well as the time decay factor controlled by system load feedback; Calculate the product of the time difference and the time decay factor, and use the negative of the product as the exponent to construct an exponential decay term; Calculate the ratio of the query frequency information to the historical peak frequency to obtain the frequency gain term; The access probability of the target data block is obtained by multiplying the frequency gain term by the first basic weight coefficient and adding the product of the exponential decay term by the second basic weight coefficient.

4. The method as described in claim 1, characterized in that, The step of obtaining the data block identifier of the target data block preloaded into the memory buffer pool to form a resident state bitmap includes: A contiguous memory space is allocated in the memory buffer pool as a global bitmap region; In response to the notification that the target data block has been asynchronously loaded, the physical page number corresponding to the data block identifier is extracted; The physical page number is mapped to a specific bit within the global bitmap area using a hash mapping function, and the specific bit is set to a valid state to generate the resident state bitmap.

5. The method as described in claim 1, characterized in that, The step of rewriting the initial execution plan according to the resident state bitmap, splitting a single scan operator into a first scan branch facing the memory buffer pool and a second scan branch facing unloaded disk data, and then sending the rewritten execution plan for execution includes: Extract the set of block indices of the physical blocks to be scanned in the single scan operator; Perform a bitwise AND operation between the block index set and the resident state bitmap to separate the first block index subset in the resident state and the second index subset in the non-resident state; A hash addressing operator is generated for the first index subset as the first scan branch, and a physical disk cursor read operator is generated for the second index subset as the second scan branch; A joint reorganization operator is inserted at the top level of the first scan branch and the second scan branch to merge the data streams and form the rewritten execution plan for execution.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: Periodically calculate the actual memory hit rate of query tasks in the memory buffer pool within a preset time period; Calculate the deviation between the actual memory hit rate and the expected hit rate; The trigger threshold used when calculating the access probability is dynamically adjusted based on the deviation value in a closed loop.

7. The method as described in claim 6, characterized in that, The trigger threshold used when calculating the access probability based on the deviation value in dynamic closed-loop adjustment includes: When the actual memory hit rate is less than the first preset hit rate threshold and the buffer pool memory balance is greater than the preset capacity threshold, it is determined that the system is in an underloaded state. The trigger threshold is then lowered according to a preset step size ratio to increase the number of asynchronous data blocks loaded. When the actual memory hit rate is less than the second preset hit rate threshold and the buffer pool memory balance is less than the preset capacity threshold, it is determined that the system is in an overload jitter state. The trigger threshold is then increased by an increment of the exponential backoff algorithm to suppress the loading of invalid data blocks. Wherein, the first preset hit rate threshold is less than the second preset hit rate threshold.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored on the memory, wherein the processor, when executing the program, implements the method of claim 1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of claim 1.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of claim 1.

Images