A data pre-reading method, device, equipment and medium

Abstract

The application discloses a data pre-reading method, device and equipment and a medium. The data pre-reading method comprises the following steps: based on first pre-reading scheduling parameters preset according to a first pre-reading scheduling period, a first pre-reading task in the first pre-reading scheduling period is executed, and a corresponding first pre-reading data set is obtained; according to the first pre-reading data set, first execution state information related to the execution of the first pre-reading task is obtained; according to the first execution state information, the first pre-reading scheduling parameters are adjusted, and second pre-reading scheduling parameters of a second pre-reading scheduling period are obtained; and according to the second pre-reading scheduling parameters, a second pre-reading task in the second pre-reading scheduling period is executed. The method can execute the first pre-reading task in the first pre-reading scheduling period, obtain the first execution state information, adjust the first pre-reading scheduling parameters, obtain the second pre-reading scheduling parameters, and thus realize more accurate and efficient data pre-reading, guarantee system performance and maintain optimal resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data pre-reading method, apparatus, device, and medium. Background Technology

[0002] In modern database systems, I / O latency is a key bottleneck restricting performance. Data block prefetching technology alleviates this problem by predicting and preloading data into memory. However, with increasing business complexity and data volume, traditional prefetching strategies face three major challenges: First, it's difficult to balance real-time response with prediction accuracy, easily leading to resource waste; second, resource contention is prominent, with prefetching's memory and bandwidth potentially impacting current business operations; and third, static parameters cannot adapt to dynamic environments such as business peaks and valleys and sudden traffic surges, resulting in rigid timing decisions (e.g., insufficient prefetching during peak periods and excessive prefetching during off-peak periods) and blind target selection (excessive prefetching of cold data and omission of hot data). Therefore, existing data block prefetching easily leads to resource control imbalances, affecting normal business operations. Summary of the Invention

[0003] This invention provides a data pre-reading method, apparatus, device, and medium to solve the problem of resource control imbalance in existing data block pre-reading, which affects normal business operations.

[0004] Firstly, a data pre-reading method is provided, comprising the steps of: executing a first pre-reading task within the first pre-reading scheduling period based on a first pre-reading scheduling parameter preset in the first pre-reading scheduling period, and obtaining a corresponding first pre-reading dataset, wherein the first pre-reading scheduling parameter includes a first distribution frequency of the first pre-reading task; obtaining first execution status information related to the execution status of the first pre-reading task based on the first pre-reading dataset, wherein the first execution status information includes a queue growth rate corresponding to a first pre-reading task queue composed of multiple first pre-reading tasks; adjusting the first pre-reading scheduling parameter based on the first execution status information to obtain a second pre-reading scheduling parameter for a second pre-reading scheduling period, wherein the second pre-reading scheduling parameter includes a second distribution frequency of the second pre-reading task within the second pre-reading scheduling period; and executing the second pre-reading task based on the second pre-reading scheduling parameter.

[0005] Secondly, a data pre-reading device is provided, comprising: a first pre-reading module, configured to execute a first pre-reading task within the first pre-reading scheduling period based on a first pre-reading scheduling parameter preset in the first pre-reading scheduling period, and obtain a corresponding first pre-reading dataset, wherein the first pre-reading scheduling parameter includes a first delivery frequency of the first pre-reading task; an information acquisition module, configured to acquire first execution status information related to the execution status of the first pre-reading task based on the first pre-reading dataset, wherein the first execution status information includes a queue growth rate corresponding to a first pre-reading task queue composed of multiple first pre-reading tasks; a parameter adjustment module, configured to adjust the first pre-reading scheduling parameter according to the first execution status information to obtain a second pre-reading scheduling parameter for a second pre-reading scheduling period, wherein the second pre-reading scheduling parameter includes a second delivery frequency of the second pre-reading task within the second pre-reading scheduling period; and a second pre-reading module, configured to execute the second pre-reading task according to the second pre-reading scheduling parameter.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data prefetching method.

[0007] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described data prefetching method.

[0008] In the above-mentioned technical solutions for data pre-reading methods, apparatus, computer equipment, and storage media, the data pre-reading method includes the following steps: 1) Executing a first pre-reading task within the first pre-reading scheduling period based on a first pre-reading scheduling parameter preset in the first pre-reading scheduling period, and obtaining a corresponding first pre-reading dataset; the first pre-reading scheduling parameter includes a first distribution frequency of the first pre-reading task; 2) Obtaining first execution status information related to the execution status of the first pre-reading task based on the first pre-reading dataset; the first execution status information includes a queue growth rate corresponding to a first pre-reading task queue composed of multiple first pre-reading tasks; 3) Adjusting the first pre-reading scheduling parameter based on the first execution status information to obtain a second pre-reading scheduling parameter for the second pre-reading scheduling period; the second pre-reading scheduling parameter includes a second distribution frequency of the second pre-reading task within the second pre-reading scheduling period; 4) Executing a second pre-reading task based on the second pre-reading scheduling parameter. This method executes the first pre-read task within the first pre-read scheduling cycle using the first pre-read scheduling parameter to obtain multi-dimensional first execution status information, and then dynamically optimizes the first pre-read scheduling parameter to obtain the second pre-read scheduling parameter, thereby achieving more accurate and efficient data pre-reading within the second pre-read scheduling cycle. This closed-loop adaptive mechanism realizes the adaptive closed-loop control of the pre-read strategy, maintaining optimal resource utilization efficiency while ensuring system performance. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is an overall flowchart of the data pre-reading method provided by the present invention; Figure 2 This is a specific flowchart of step S3 in the data pre-reading method provided in Embodiment 1 of the present invention; Figure 3 This is a specific flowchart of step S3 in the data pre-reading method provided in Embodiment 2 of the present invention; Figure 4 This is a specific flowchart of step S3 in the data pre-reading method provided in Embodiment 3 of the present invention; Figure 5 This is a specific flowchart of step S3 in the data pre-reading method provided in Embodiment 4 of the present invention; Figure 6 This is a schematic diagram of a data pre-reading device in one embodiment of the present invention; Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] It should be noted first that some key terms involved in this application and their corresponding explanations are as follows: Data caching: A technology that temporarily stores frequently accessed or computationally expensive data in high-speed storage media such as memory and SSDs. Its core logic is to allow data to be accessed "nearest", thereby breaking the performance bottleneck of slow storage (such as mechanical hard drives and remote databases).

[0013] Asynchronous read-ahead: A technique to optimize data reading efficiency. The system analyzes the current data access pattern (such as sequential reading, high-frequency access) to predict the data that may be needed later, and starts a read-ahead operation independently in the background. This data is loaded in advance from slow storage media such as disks into memory buffers or caches. The entire read-ahead process does not block the normal read and write operations of the current application. When the application needs to access the data later, it can directly retrieve it from high-speed storage media, thereby reducing disk seek time and I / O wait time, improving system response speed and overall throughput. It is commonly used in scenarios such as database queries, large file read and write, and streaming media transmission.

[0014] like Figure 1 As shown, this application provides a data pre-reading method, including the following steps: Step S1: Based on the first pre-read scheduling parameters preset in the first pre-read scheduling period, execute the first pre-read task within the first pre-read scheduling period and obtain the corresponding first pre-read dataset.

[0015] It should be noted that the first prefetch scheduling period is a system-preset time interval used to trigger the first prefetch operation. Its value is dynamically adjusted based on the current load status, historical access patterns, and storage media characteristics to ensure an optimal balance between resource consumption and prefetch benefits. The first prefetch scheduling parameters are the core configurations that determine the amount of data to be prefetched, the prefetch depth, and the cache priority. They are a combination of parameters that match the first prefetch scheduling period, including the first issuance frequency of the first prefetch task, the first issuance amount of the first data to be prefetched in the first prefetch task, the cache residence time of the first prefetch dataset, and differentiated prefetch granularity set for different data heat levels of the first data to be prefetched in the first prefetch task. The first prefetch task is the dataset of the first data to be prefetched within the first prefetch scheduling period. Its execution process strictly follows parameter constraints, prioritizing the loading of hot data segments while ensuring low system overhead, thus building a response buffer layer for subsequent accesses. The first pre-read dataset is a cached data set consisting of the first data to be pre-read after the execution of the corresponding first pre-read task. Its structure is organized hierarchically according to access popularity and temporal locality, and includes metadata index, data block hash fingerprint and pre-read confidence label, etc., to ensure that subsequent queries can quickly locate high-value data blocks based on the popularity label, while supporting deduplication verification based on hash fingerprint and millisecond-level retrieval of metadata index.

[0016] like Figure 2As shown, in Embodiment 1, a first pre-read task is first issued according to a first issuance frequency. The first pre-read task includes multiple first pre-read data. Then, the first pre-read task is executed, and each first pre-read data is loaded into the memory cache in batches according to the popularity level and pre-read granularity requirements. After each batch is loaded, the hash fingerprint and pre-read confidence label of the corresponding data block are updated in real time, and the metadata index is refreshed synchronously. When the first pre-read scheduling cycle ends, part of the first pre-read data in the first pre-read task is pre-read into the memory cache to form a structured first pre-read dataset. The remaining first pre-read data in the first pre-read task is accumulated to the next pre-read scheduling cycle as the input basis for the data pool to be scheduled to enter the second pre-read scheduling cycle.

[0017] Step S2: Based on the first pre-read dataset, obtain the first execution status information related to the execution status of the first pre-read task.

[0018] It should be noted that the first execution status information includes status indicators such as the queue growth rate, foreground loading probability, buffer utilization rate, pre-read hit rate, and data access popularity of the data to be pre-read. Among them, the queue growth rate reflects the task backlog speed, the foreground loading probability represents the tendency of pre-read data to be called immediately, the buffer utilization rate reveals the cache space pressure level, the pre-read hit rate directly reflects the effectiveness of the pre-read strategy, and the data access popularity is calculated by weighting the actual access frequency within the sliding window with a time decay factor. The above-mentioned indicators can be used individually or in combination to quantify the execution efficiency and resource adaptability of the pre-read task, and together they constitute the core input of the dynamic feedback closed loop.

[0019] In Example 1, the first execution status information includes the foreground loading probability and buffer usage rate corresponding to the first pre-read task.

[0020] It should be noted that in asynchronous prefetching, both situations—prefetching too quickly leading to premature page eviction and prefetching scheduling being too slow causing the page to be loaded directly from the foreground—result in the foreground loading directly instead of hitting the prefetch cache, leading to resource waste. Therefore, Implementation Example 1 introduces the foreground loading probability, which is the proportion of data directly loaded from the foreground to the total data requested per unit time. The lower the value, the more sufficient the prefetch coverage.

[0021] The data loaded directly in the foreground includes the first data to be pre-read that did not hit the pre-read cache during the first pre-read scheduling cycle, as well as the data blocks loaded directly from the disk by the foreground thread; the total load request data includes the pre-read hit data (i.e. the first data to be pre-read that was pre-read in the first pre-read data set) and the data loaded directly in the foreground.

[0022] Furthermore, based on the first pre-read dataset, the original log records of the data directly loaded in the foreground and the total data loaded in the total load request are obtained, and then the data attribution is completed by timestamp alignment and hash verification; then the foreground loading probability can be obtained by dividing the amount (or number) of data directly loaded in the foreground by the amount (or number) of data loaded in the total load request.

[0023] Furthermore, the system acquires real-time data on directly loaded data and total load requests, and calculates the front-end load probability within a time window (e.g., 1 second or a pre-read cycle). For example, if there are 100 total load requests in a certain cycle, and 20 of them are directly loaded from the front end, then the front-end load probability is 20%.

[0024] Meanwhile, Implementation Example 1 also incorporates buffer utilization rate, which is the ratio of the current buffer's used capacity to the total capacity, to dynamically assess the stress level of cache resources.

[0025] The process begins by obtaining the real-time status of the buffer through the memory management module, including the used buffer capacity and the total buffer capacity. The used buffer capacity is the memory space occupied by pre-read data, user request data, etc.; the total buffer capacity is the maximum memory limit allocated by the system for pre-read tasks. Then, the buffer utilization rate is obtained by calculating the ratio of the used buffer capacity to the total buffer capacity in real time. For example, if the total buffer capacity is 10GB and the currently used buffer capacity is 8GB, the buffer utilization rate is 80%.

[0026] Therefore, in Implementation Example 1, data is collected at a millisecond frequency by an independent monitoring thread to obtain real-time observations of the foreground loading probability and buffer usage rate, ensuring that the indicators reflect the current state of the system and providing a quantitative basis for the dynamic adjustment of the subsequent second pre-read scheduling parameters.

[0027] Step S3: Adjust the first pre-read scheduling parameters according to the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period.

[0028] It should be noted that the second pre-read scheduling cycle is a completely new scheduling phase initiated after the first pre-read scheduling cycle ends, ensuring the continuity and evolution of data pre-read scheduling. The second pre-read scheduling parameters inherit from the first pre-read scheduling parameters, and dynamically adjust parameters such as the corresponding first distribution frequency, the cache dwell time of the first pre-read dataset, and the differentiated pre-read granularity set for different data heat levels of the first data to be pre-read in the first pre-read task. This yields parameters such as the second distribution frequency, the second distribution amount, the cache dwell time of the second pre-read dataset, and the differentiated pre-read granularity set for different data heat levels of the second data to be pre-read in the second pre-read task that need to be executed in the corresponding second pre-read scheduling cycle.

[0029] In Example 1, as Figure 2 As shown, step S3 includes the following sub-steps: Step S301: Obtain the foreground loading probability.

[0030] Step S302: Determine whether the probability of loading the front end is less than the preset probability threshold.

[0031] If the probability of loading the foreground is greater than or equal to a preset probability threshold, proceed to step S303; if the probability of loading the foreground is less than the preset probability threshold, proceed to step S304.

[0032] Step S303: Reduce the first sending frequency to obtain the second sending frequency.

[0033] Step S304: Obtain the buffer usage rate.

[0034] Step S305: Determine whether the buffer utilization rate is less than the preset utilization rate threshold.

[0035] If the buffer utilization rate is greater than or equal to the preset utilization rate threshold, proceed to step S303; if the buffer utilization rate is less than the preset utilization rate threshold, proceed to step S306.

[0036] Step S306: Increase the first sending frequency to obtain the second sending frequency.

[0037] The above steps involve dynamically adjusting the pre-reading scheduling strategy by statistically analyzing the foreground loading probability after the first pre-reading task is issued: if the foreground loading probability is greater than or equal to a preset probability threshold, it indicates that the pre-reading effectiveness is low, and the first issuance frequency of the first pre-reading task can be reduced; if the foreground loading probability is less than the preset probability threshold, it indicates that the pre-reading effect is good, and the first issuance frequency of the first pre-reading task can be appropriately increased.

[0038] Meanwhile, the buffer pool load status is monitored in real time. When the buffer utilization rate is detected to reach the preset threshold, the pre-read frequency is automatically reduced or the pre-read operation is paused, regardless of the current pre-read effectiveness, to avoid pre-read occupying too much buffer resources, further optimize the adaptive adjustment effect of the first issuance frequency, and balance the pre-read effectiveness and buffer resource utilization.

[0039] The adjustment range of the first delivery frequency is positively correlated with the deviation of the foreground loading probability and the buffer utilization rate, ensuring that the pre-read behavior not only responds to the actual needs of users, but also avoids the waste of memory resources due to excessive pre-reading, so as to obtain the second delivery frequency of the second pre-read task.

[0040] Furthermore, this embodiment uses both the foreground loading probability and buffer utilization rate as the basis for adjusting the first delivery frequency, forming a two-dimensional feedback loop: the former points to the accurate capture of user behavior intent, while the latter anchors the rational constraints of system resource boundaries. The two work together to provide a more robust decision-making basis for the dynamic optimization of the first delivery frequency, ensuring the accurate generation of the second delivery frequency. For example, when the foreground loading probability is low (good pre-reading effect) but the buffer utilization rate is high, it is still necessary to reduce the first delivery frequency to obtain the second delivery frequency, in order to avoid resource contention.

[0041] In other embodiments, a two-dimensional weighted evaluation model can be constructed by fusing the foreground loading probability and buffer utilization rate, with the foreground loading probability set as the weight coefficient. Buffer utilization rate is set as a weighting factor. Real-time calculation of overall deviation ,when The parameters are recalibrated in real time; the threshold benchmark is dynamically updated based on historical sliding window data, so that the second pre-read scheduling parameter maintains the optimal balance between millisecond-level response accuracy and resource consumption under real-time load.

[0042] In other embodiments, either the foreground loading probability or the buffer utilization rate can be used alone to dynamically adjust the first prefetch scheduling parameter to obtain the second prefetch scheduling parameter. For example, the dynamic adjustment of the first prefetch scheduling parameter can be triggered only when the foreground loading probability exceeds a probability threshold (e.g., 30%), or only when the buffer utilization rate exceeds a limit (e.g., ≥95%), automatically reducing the first delivery frequency or pausing the prefetch operation to avoid excessive occupation of system resources. Simultaneously, the second prefetch scheduling parameter can converge to a safe threshold range in real time according to a preset gradient step size. Although this single-dimensional strategy simplifies implementation, it is suitable for scenarios with clear resource constraints or highly stable user behavior, balancing response efficiency and system robustness.

[0043] Step S4: Execute the second pre-read task within the second pre-read scheduling period according to the second pre-read scheduling parameters.

[0044] It should be noted that the second pre-read task is the set of data to be pre-read within the second pre-read scheduling cycle. Its execution process strictly follows the distribution frequency, cache dwell time, and heat-based granularity strategy defined by the second pre-read scheduling parameters. The second pre-read task specifically includes newly added second pre-read data and the migrated data from the incomplete portion of the first pre-read task, ensuring the continuity and consistency of pre-reading.

[0045] In Example 1, according to the second issuance frequency of the second pre-read task obtained in step S3, the corresponding second pre-read task is issued. The second pre-read task includes multiple second pre-read data. The second pre-read task is then executed, and each second pre-read data is loaded into the memory cache in batches according to the popularity level and pre-read granularity requirements updated by the first pre-read scheduling cycle. After each batch is loaded, the hash fingerprint and pre-read confidence label of the corresponding data block are updated in real time, and the metadata index is refreshed synchronously. When the second pre-read scheduling cycle ends, part of the second pre-read data in the second pre-read task is pre-read into the memory cache to form a structured second pre-read dataset. The remaining second pre-read data in the second pre-read task is accumulated to the next pre-read scheduling cycle as the input basis for the data pool to be scheduled to enter the third pre-read scheduling cycle.

[0046] Furthermore, based on the second pre-read dataset, iterative closed-loop steps S2-S4 can be performed to continuously optimize the parameter adaptation accuracy and data pre-read performance of subsequent pre-read scheduling cycles, enabling the pre-read behavior to adaptively respond to changes in the system operating environment.

[0047] like Figure 3 As shown, in Embodiment 2, unlike the previous embodiments, this embodiment achieves adaptive adjustment of the issuance frequency of corresponding pre-read tasks by monitoring the queue growth rate of the pre-read queue length. During busy business periods, increased disk I / O pressure leads to a decrease in read speed, resulting in longer pre-read task execution times and more frequent issuance, causing a backlog of pre-read queue tasks and an increase in the queue growth rate. Conversely, during idle business periods, good disk I / O performance improves the execution efficiency of pre-read tasks, reduces task issuance, accelerates queue consumption, and correspondingly reduces the queue growth rate.

[0048] Based on this pattern, Example 2 uses an independent thread to periodically detect changes in the growth rate of the pre-read queue length: when the queue growth rate is high, it is determined to be a peak business period, and the frequency of issuing pre-read tasks is automatically reduced to alleviate the system load; when the queue growth rate is low, it is determined to be an idle business period, and the frequency of issuing pre-read tasks is automatically increased to make full use of system resources, thereby achieving dynamic optimization of the pre-read strategy.

[0049] In step S1, the first pre-read task comprises multiple tasks, forming a first pre-read task queue that constitutes the parallel pre-read job flow within the first pre-read scheduling cycle. When 100 data tables concurrently execute full table scan queries, the front-end application continuously sends data read requests for consecutive pages. The initial backlog of the first pre-read task queue is 100 units, while the thread quota of the pre-read thread pool is only 10. Due to the high query load, the background pre-read thread and the foreground query thread compete for IO resources, leading to disk IO channel congestion and a significant decrease in data read throughput. Consequently, within the first pre-read scheduling cycle, the pre-read thread only completes the consumption of 30 first pre-read tasks, obtaining the first pre-read dataset corresponding to those 30 tasks. Simultaneously, the unprocessed 70 first pre-read tasks trigger the foreground thread to degrade to direct disk reads, resulting in an increase in average query latency. At the same time, the 100 newly issued second pre-read tasks from the front end, together with the 70 unprocessed first pre-read tasks in the first pre-read task queue, form 170 second pre-read tasks, which constitute the second pre-read task queue. The queue growth rate reaches 70 tasks / second, far exceeding the instantaneous processing capacity limit of the pre-read thread pool, and triggers the second pre-read scheduling cycle.

[0050] Furthermore, each first pre-read task dynamically allocates scheduling resources based on data popularity weight and IO path priority to ensure that high-confidence pre-read requests receive lower latency responses; the first pre-read queue adopts a double buffer mechanism, which continuously updates task metadata and popularity model in the background while performing pre-read in the foreground, realizing millisecond-level adaptive evolution of scheduling strategies.

[0051] In step S2, based on the number of the first pre-read datasets, the number of unprocessed first pre-read tasks in the first pre-read task queue can be calculated, and the queue growth rate corresponding to the first pre-read task queue can be obtained, which is 70 tasks / second. This queue growth rate is completely consistent with the actual growth rate of the second pre-read task queue, verifying the quantitative capture capability of the pre-read scheduling model for sudden loads.

[0052] For step S3, as Figure 3 As shown, step S3 includes the following sub-steps: Step S311: Obtain the queue growth rate.

[0053] Step S312: Determine whether the queue growth rate is less than the preset growth rate threshold.

[0054] If the queue growth rate is greater than or equal to the preset growth rate threshold, proceed to step S313; if the queue growth rate is less than the preset growth rate threshold, proceed to step S314.

[0055] Step S313: Determine that this is a peak business period, reduce the first sending frequency, and obtain the second sending frequency.

[0056] Step S314: Determine that this is a business idle period, and increase the first sending frequency to obtain the second sending frequency.

[0057] Specifically, when the queue growth rate is greater than or equal to a preset growth rate threshold, the first dispatch frequency is reduced to obtain the second dispatch frequency for the second pre-read task. When the queue growth rate is less than the growth rate threshold, the first dispatch frequency is increased to obtain the second dispatch frequency for the second pre-read task, thus ensuring continuous optimization of the subsequent pre-read scheduling closed loop.

[0058] Furthermore, based on the real-time collected queue growth rate of 70 tasks / second, it is determined that the current IO load exceeds the limit, and the pre-reading frequency dynamic adjustment mechanism is activated to reduce the issuance frequency of the second pre-reading task in the second pre-reading scheduling cycle to 50 tasks / second in order to reduce the system load.

[0059] In step S4, the second pre-read task also includes multiple tasks, forming a second pre-read task queue. This queue consists of 100 newly added pre-read tasks and 70 unpre-read tasks from the first pre-read task queue, totaling 170 tasks, constituting a complete scheduling unit. This second pre-read task queue is continuously executed at a dispatch frequency of 50 tasks / second. During this process, if severe backlog or scheduling delays are detected in the second pre-read task queue, causing the target page to be loaded into the buffer buffer by the foreground when the second pre-read task is executed (i.e., pre-read resources reach a bottleneck), the dispatch frequency of the second pre-read task is simultaneously reduced to 40 tasks. Subsequently, the pre-read task queue grows from 140 to 190 tasks, corresponding to a decrease in the queue growth rate of the second pre-read task to 20 tasks / second, but still showing positive growth. Therefore, the dispatch frequency is further slightly reduced to 30 tasks / second, ultimately achieving a balanced "dispatch-consumption" state (30 tasks / second dispatch, 30 tasks / second consumption), avoiding excessive queue backlog that could exacerbate IO blocking.

[0060] Furthermore, after a partial full table query is completed, the disk I / O load is released, and the consumption rate of the corresponding pre-read tasks exceeds the issuance rate threshold. This results in a negative growth trend in the pre-read task queue length, with the pre-read tasks maintaining a state where they are completed before the front-end page requests. Upon detecting this change, the system reverses this trend by increasing the issuance frequency of the corresponding pre-read tasks to ensure a precise match between the pre-read data supply and the remaining query demands, maintaining a dynamic balance between pre-read efficiency and I / O resource utilization.

[0061] like Figure 4 As shown, in Embodiment 3, unlike the previous embodiments, this embodiment dynamically adjusts the first amount of data to be sent to the first pre-reading task based on the pre-reading hit rate of the first data to be sent to the first pre-reading task, so as to obtain the second amount of data to be sent to the second pre-reading task. Its adjustment logic forms a closed-loop feedback with the first amount of data sent.

[0062] It should be noted that the first data to be pre-read refers to data blocks that have not yet been pre-read within the first pre-read scheduling cycle but have entered the pre-read queue. Its initial distribution is jointly constrained by the current cache hit rate and the locality of access characteristics in the foreground. The second data to be pre-read, on the other hand, refers to potentially hot data blocks predicted based on the foreground access pattern within the second pre-read scheduling cycle. Its second distribution not only inherits the adjustment results of the first distribution but also incorporates the pre-read hit rate feedback value of the first data to be pre-read, ensuring that the pre-read decision is always anchored at the critical point between data value and system overhead.

[0063] like Figure 4 As shown, unlike the previous embodiments, step S3 in this embodiment includes the following sub-steps: Step S321: Obtain the pre-read hit rate.

[0064] Among them, the pre-read hit rate serves as a core feedback indicator, reflecting in real time the degree of matching between the pre-read data and the actual access needs of the front end.

[0065] Step S322: Determine whether the pre-read hit rate is less than the preset hit rate threshold.

[0066] If the queue growth rate is greater than or equal to the preset hit rate threshold, proceed to step S323; if the queue growth rate is less than the preset hit rate threshold, proceed to step S324.

[0067] Step S323: Increase the first distribution amount to obtain the second distribution amount.

[0068] Step S324: Reduce the first amount of data sent to obtain the second amount of data sent.

[0069] In step S1, the initial set number of consecutive pages sent at one time (i.e., the first number of data to be pre-read) is 100, corresponding to the first pre-read task of pre-reading 100 consecutive pages. After executing the first pre-read task in step S1, a first pre-read dataset consisting of multiple consecutive pages is obtained. Then, in steps S2-S3, if the pre-read hit rate of the front-end application layer query for the pre-read pages is greater than or equal to a preset hit rate threshold, a pre-read window expansion strategy is triggered, increasing the first number of pages sent at one time to 120, resulting in a second number of data to be pre-read in the second pre-read task of the next pre-read scheduling cycle of 120. Conversely, if the first number of pages sent is less than 120, the second number of pages sent is reduced to obtain the second number of pages sent. In step S4, based on the second number of pages sent, a second pre-read task is executed, generating a second pre-read dataset consisting of more consecutive pages. The process returns to step S2, replacing the first pre-read dataset with the second pre-read dataset, forming a closed-loop feedback mechanism.

[0070] Furthermore, after multiple iterations, if the pre-read hit rate of the front-end application layer query for the pre-read page remains high, a pre-read window expansion strategy is triggered, such as increasing the number of consecutive pages sent in a single batch to 120. If the pre-read hit rate remains stable in the high threshold range after expansion, incremental expansion continues; once a significant decline in the pre-read hit rate is detected, the previous effective configuration (e.g., 120 pages) is automatically locked as the current equilibrium point, and the expansion process is terminated.

[0071] Furthermore, when sudden fluctuations in business load intensify competition for IO resources, leading to extended processing latency for pre-read tasks, and when high-frequency pre-read blocking occurs in front-end queries (i.e., the foreground thread continuously waits for the background pre-read completion signal), the current pre-read granularity is determined to exceed the resource carrying capacity threshold by real-time monitoring of blocking frequency and IO response indicators. Then, a shrinkage mechanism is initiated to revert the single continuous page distribution volume (e.g., to 80 pages).

[0072] The entire process of Example 3 relies on pre-read hit rate feedback and resource load awareness to achieve closed-loop adaptive adjustment of the pre-read window size, maximizing pre-read coverage efficiency while ensuring dynamic adaptation to system IO processing capabilities and business query characteristics.

[0073] In some embodiments, the prefetch window expansion strategy can be triggered after multiple rounds of prefetch scheduling iterations. Specifically, when the prefetch hit rate remains consistently high across multiple consecutive prefetch scheduling cycles, the system determines that the access pattern is stabilizing and increases the amount of consecutive pages sent per cycle. During this period, each prefetch scheduling cycle maintains the initial amount of consecutive pages sent per cycle. Simultaneously, the system automatically records the prefetch hit rate fluctuation trajectory for each cycle and uses a sliding window algorithm to eliminate instantaneous noise, ensuring that the strategy triggering conditions are statistically significant. For example, the prefetch window expansion strategy is only executed if the prefetch hit rate is ≥95% for three consecutive cycles.

[0074] like Figure 5As shown in Example 4, unlike the previous examples, Example 4 provides an asynchronous pre-read intelligent decision-making scheme based on page access popularity. First, a page hot / cold grading statistical model is constructed. This model tracks the number of accesses, time distribution, and frequency of consecutive accesses throughout the page's lifecycle, from loading into the buffer to being evicted. Hot and cold grading standards are established by combining access frequency, time distribution characteristics, data block storage characteristics, and access patterns, and the popularity tags are updated periodically to adapt to business changes. Then, because hot data in the buffer has high access frequency and short-term decay characteristics, it is accessed relatively frequently in the recent period, and is often accessed consecutively, with a high proportion of consecutive accesses. Cold data, on the other hand, exhibits low frequency, long period, and discrete access characteristics. The buffer is hit less frequently, and there is no frequent access in the recent period, nor is there any consecutive access, with a high proportion of discrete accesses. Therefore, a differentiated pre-read strategy is implemented for the two: hot data is pre-read in large granular batches, while cold data is pre-read in fine-grained small-range formats, executed only when the buffer is under low pressure. This improves pre-read efficiency while reducing unnecessary IO overhead.

[0075] Meanwhile, by monitoring the front-end read waiting status in real time to determine the IO load status, bidirectional granular adjustment is performed on hot and cold data respectively, and cold data pre-reading is synchronously linked to buffer pressure. When the pressure is high, it is directly terminated to prioritize the pre-reading resources of hot data, thus realizing intelligent adaptation of the pre-reading strategy in all aspects.

[0076] For step S3, as Figure 5 As shown, unlike the previous embodiments, step S3 in this embodiment includes the following sub-steps: Step S331: Determine the data popularity level of the second data to be pre-read based on the data access popularity.

[0077] It should be noted that, based on the description of the foregoing embodiments, the second data to be pre-read includes the first data to be pre-read that has not been pre-read and newly added data to be pre-read. This data can be categorized into two types based on its access frequency: hot data and cold data. Each type of data corresponds to a different pre-read granularity. The data popularity levels include a first popularity level and a second popularity level, with the first popularity level corresponding to hot data and the second popularity level corresponding to cold data.

[0078] Step S332: Based on the data popularity level, adjust the first pre-read granularity corresponding to different access popularity to obtain the second pre-read granularity corresponding to different access popularity.

[0079] Specifically, when the data popularity level is the first popularity level, the first pre-read granularity corresponding to the first popularity level is increased to obtain the second pre-read granularity. When the data popularity level is the second popularity level and the buffer utilization rate is not lower than the utilization rate threshold, the first pre-read granularity corresponding to the second popularity level is decreased to obtain the second pre-read granularity.

[0080] The process involves setting the first pre-read granularity based on historical data: 100 consecutive pages are initially pre-read for hot data, and 20 discrete pages are initially pre-read for cold data. After executing steps S1-S2, the data popularity level of the second data to be pre-read is determined based on real-time data access frequency. Then, the first pre-read granularity corresponding to different access frequencies is dynamically adjusted according to the data popularity level to obtain the second pre-read granularity required for the second pre-read scheduling cycle. For hot data corresponding to the first popularity level, due to frequent user queries and no waiting events on the front end, the pre-read granularity is adjusted to 140 consecutive pages in increments of 20 pages, thus maximizing IO bandwidth utilization. For cold data corresponding to the second popularity level, if the buffer utilization rate reaches 80% (high pressure) during cold data pre-reading, the corresponding second pre-read granularity is reduced to 10 discrete pages to avoid crowding out hot data space.

[0081] Therefore, Example 4 establishes a data popularity tag system (hot data / cold data) by performing hit statistics on the entire lifecycle of a page from loading into the buffer to being evicted; combined with the buffer load status, the pre-read strategy is dynamically adjusted, prioritizing the pre-read of hot data to focus on core resources when the load is high, and simultaneously covering hot and cold data when the load is low to improve overall access efficiency, thereby achieving differentiated and precise allocation of pre-read resources.

[0082] In some embodiments, based on Embodiment 4, subsequent statistics revealed that the pre-read hit rate of hot data was only 65%, indicating that the second pre-read granularity was too large. Therefore, by gradually reducing 10 pages each time, the pre-read hit rate was gradually adjusted to 110 pages, and finally the pre-read hit rate stabilized.

[0083] It should be noted that the above embodiments only construct multi-granularity, scenario-specific, and adaptive pre-read scheduling strategies from different dimensions. In actual use, the above strategies can work together. For example, the load status of the buffer (i.e., the BufferPool) can be monitored in real time. When the buffer utilization rate reaches a preset utilization threshold, the pre-read frequency is automatically reduced or the pre-read operation is paused to avoid excessive occupation of system resources. At the same time, a front-end loading and pre-read hit feedback mechanism is established. For situations where a pre-read task is requested by the front end before completion or the pre-read data is loaded but not used in time and is thus eliminated, the scheduling strategy and timing of pre-read tasks are continuously optimized by statistically analyzing the probability distribution of front-end page loading and pre-read hits. Furthermore, an asynchronous pre-read mode can be adopted while ensuring that the front-end business response is not affected. At the same time, a task load monitoring mechanism is established. Based on the backlog of the pre-read task queue and the IO performance fluctuations caused by peak business periods, the pre-read execution frequency and concurrency are dynamically adjusted to achieve reasonable allocation of system resources.

[0084] In summary, this application discloses a data prefetching method. This method dynamically adjusts the prefetching frequency, page type, and prefetching granularity in real time based on multi-dimensional information such as system IO load, buffer pressure, prefetch hit statistics, queue change trends, and page hot / cold analysis. This ensures optimal resource utilization efficiency while maintaining system performance, enabling the prefetching behavior to adaptively respond to changes in the system operating environment. Specifically, based on multi-dimensional parameters such as system resource load and environmental pressure, a dynamic adjustment model can be constructed by combining the prefetch hit rate with the task queue running status (such as queue growth rate) to achieve adaptive calibration of the prefetching frequency, ensuring that the prefetching rhythm is precisely matched with IO processing capacity and business query intensity. Simultaneously, relying on real-time collected resource utilization and environmental pressure indicators, a multi-dimensional decision-making algorithm dynamically adjusts the prefetching window size (single prefetch data volume). When resources are abundant, the prefetching range is expanded to improve coverage efficiency; when resources are scarce, the prefetching granularity is contracted to avoid resource contention. By deeply integrating business access patterns, real-time system status, and hardware performance characteristics, a dynamic and adaptive prefetching decision mechanism is constructed to maximize the role of prefetching.

[0085] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0086] In one embodiment, a data prefetching device is provided, which corresponds one-to-one with the data prefetching method in the above embodiments. For example... Figure 6 As shown, the data pre-reading device includes a first pre-reading module 101, an information acquisition module 102, a parameter adjustment module 103, and a second pre-reading module 104. Detailed descriptions of each functional module are as follows: The first pre-read module 101 is used to execute the first pre-read task within the first pre-read scheduling period based on the first pre-read scheduling parameters preset in the first pre-read scheduling period, and obtain the corresponding first pre-read dataset.

[0087] The information acquisition module 102 is used to acquire first execution status information related to the execution status of the first pre-reading task based on the first pre-reading dataset.

[0088] The parameter adjustment module 103 is used to adjust the first pre-read scheduling parameters according to the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period.

[0089] The second pre-read module 104 is used to execute the second pre-read task within the second pre-read scheduling period according to the second pre-read scheduling parameters.

[0090] Specific limitations regarding the data prefetching device can be found in the limitations of the data prefetching method described above, and will not be repeated here. Each module in the aforementioned data prefetching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0091] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data prefetching method.

[0092] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data prefetching method described in the above embodiment, for example... Figure 1 As shown in S1-S4, or Figures 2 to 5 As shown, to avoid repetition, it will not be described again here. Alternatively, the processor executes a computer program to implement the functions of each module / unit in this embodiment of the data prefetching device, for example... Figure 6 The functions of the first pre-read module 101, information acquisition module 102, parameter adjustment module 103, and second pre-read module 104 shown are not described again here to avoid repetition.

[0093] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the data prefetching method described in the above embodiment, for example... Figure 1 As shown in S1-S4, or Figures 2 to 5 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the data prefetching device, for example... Figure 6 The functions of the first pre-read module 101, information acquisition module 102, parameter adjustment module 103, and second pre-read module 104 shown are not described again here to avoid repetition. The computer-readable storage medium can be non-volatile or volatile.

[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0095] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0096] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0097] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data pre-reading method, characterized in that, Including the following steps: Based on the first pre-read scheduling parameters preset in the first pre-read scheduling period, the first pre-read task within the first pre-read scheduling period is executed, and the corresponding first pre-read dataset is obtained. The first pre-read scheduling parameters include the first distribution frequency of the first pre-read task. Based on the first pre-read dataset, obtain first execution status information related to the execution status of the first pre-read task. The first execution status information includes the queue growth rate corresponding to the first pre-read task queue composed of multiple first pre-read tasks. Based on the first execution status information, the first pre-read scheduling parameters are adjusted to obtain the second pre-read scheduling parameters for the second pre-read scheduling period. The second pre-read scheduling parameters include the second issuance frequency of the second pre-read task within the second pre-read scheduling period. The second pre-read task is executed according to the second pre-read scheduling parameters.

2. The data pre-reading method according to claim 1, characterized in that, The step of adjusting the first pre-read scheduling parameters according to the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period includes: When the queue growth rate is greater than or equal to a preset growth rate threshold, the first sending frequency is reduced to obtain the second sending frequency of the second pre-read task; When the queue growth rate is less than the growth rate threshold, the first sending frequency is increased to obtain the second sending frequency.

3. The data pre-reading method according to claim 1, characterized in that, The first execution status information also includes the foreground loading probability. The step of adjusting the first pre-read scheduling parameters based on the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period includes: When the foreground loading probability is less than a preset probability threshold, the first sending frequency is increased to obtain the second sending frequency; When the foreground loading probability is greater than or equal to the probability threshold, the first sending frequency is reduced to obtain the second sending frequency.

4. The data pre-reading method according to claim 3, characterized in that, The first execution status information also includes buffer utilization. The step of adjusting the first pre-read scheduling parameters based on the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period further includes: When the buffer utilization rate is greater than or equal to a preset utilization rate threshold, the first sending frequency is reduced to obtain the second sending frequency. When the buffer usage rate is less than a preset usage rate threshold, the first sending frequency is increased to obtain the second sending frequency.

5. The data pre-reading method according to any one of claims 1-4, characterized in that, The first execution status information includes the pre-read hit rate of the first data to be pre-read in the first pre-read task, the first pre-read scheduling parameter includes the first amount of the first data to be pre-read, and the second pre-read scheduling parameter includes the second amount of the second data to be pre-read in the second pre-read task. The step of adjusting the first pre-read scheduling parameters according to the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period includes: When the pre-read hit rate is greater than or equal to a preset hit rate threshold, the first sending amount is increased to obtain the second sending amount; When the pre-read hit rate is less than the hit rate threshold, the first sending amount is reduced, and the second sending amount is obtained.

6. The data pre-reading method according to claim 1, characterized in that, The first execution status information also includes the data access popularity of the second data to be pre-read in the second pre-read task; the first pre-read scheduling parameter includes the first pre-read granularity of the first data to be pre-read with different access popularity in the first pre-read task; and the second pre-read scheduling parameter includes the second pre-read granularity of the second data to be pre-read with different access popularity. The step of adjusting the first pre-read scheduling parameter according to the first execution status information to obtain the second pre-read scheduling parameter for the second pre-read scheduling period includes: Based on the data access popularity, determine the data popularity level of the second data to be pre-read; Based on the data popularity level, the first pre-read granularity corresponding to different access popularity is adjusted to obtain the second pre-read granularity corresponding to different access popularity.

7. The data pre-reading method according to claim 6, characterized in that, The data popularity level includes a first popularity level and a second popularity level; based on the data popularity level, the first pre-read granularity corresponding to different access popularity is adjusted to obtain the second pre-read granularity corresponding to different access popularity, including: When the data popularity level is the first popularity level, the first pre-read granularity corresponding to the first popularity level is increased to obtain the second pre-read granularity; When the data popularity level is the second popularity level, the first pre-read granularity corresponding to the second popularity level is reduced to obtain the second pre-read granularity.

8. A data pre-reading device, characterized in that, include: The first pre-read module is used to execute the first pre-read task within the first pre-read scheduling period based on the first pre-read scheduling parameters preset in the first pre-read scheduling period, and obtain the corresponding first pre-read dataset. The first pre-read scheduling parameters include the first distribution frequency of the first pre-read task. The information acquisition module is used to acquire first execution status information related to the execution status of the first pre-reading task based on the first pre-reading dataset. The first execution status information includes the queue growth rate corresponding to the first pre-reading task queue composed of multiple first pre-reading tasks. The parameter adjustment module is used to adjust the first pre-read scheduling parameters according to the first execution status information to obtain the second pre-read scheduling parameters for the second pre-read scheduling period. The second pre-read scheduling parameters include the second issuance frequency of the second pre-read task within the second pre-read scheduling period. The second pre-read module is used to execute the second pre-read task according to the second pre-read scheduling parameters.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data prefetching method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data prefetching method as described in any one of claims 1 to 7.

Images