Data reading method and device, equipment and storage medium

Abstract

The invention discloses a data reading method and device, computer equipment and a storage medium. The method comprises the following steps: setting a cache layer in a memory; in response to the received first data reading request, reading target data information from a hard disk layer; selecting a data block adjacent to the position of the target data information from the hard disk layer according to the physical position of the target data information in the hard disk layer and a preloading algorithm, and preloading the data block to a cache layer; and in response to the received second data reading request, reading the target data information from the cache layer, and if the target data information does not exist in the cache layer, reading the target data information from the hard disk layer. Through the method, the data reading efficiency can be greatly improved.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data reading method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of cloud computing and virtualization technologies, hyper-converged infrastructure (HCI) has gradually become the mainstream data center architecture. HCI significantly simplifies deployment and maintenance by unifying computing, storage, and network resources in a software-defined manner, offering excellent scalability and resource utilization. Currently, mainstream HCI systems typically use SSDs or NVMe hard drives as the primary storage medium, supporting upper-layer services such as virtual machines and containers.

[0003] To improve data access speed, existing hyperconverged systems typically employ technologies such as tiered storage (separation of hot and cold data), data caching (e.g., SSD-based read / write caching), and IO scheduling optimization. For example, some systems configure an SSD cache layer in front of the NVMe drives or use a log-based write mechanism to reduce random IO pressure. Some products support using software-defined methods to use a portion of memory as a caching layer to accelerate disk IO. In addition, there are technologies that utilize data access frequency for data prefetching or use IO prediction algorithms to optimize IO scheduling strategies.

[0004] While the above methods alleviate storage performance bottlenecks to some extent, they still have significant drawbacks. The main problems include: the huge gap in IOPS, bandwidth, and latency between hard drives (even NVMe SSDs) and memory means that the storage layer remains a performance bottleneck when facing high-concurrency, low-latency data requests; secondly, most existing SSD or hard drive caching mechanisms fail to fully utilize the large memory resources in modern servers, resulting in low memory utilization; furthermore, traditional cache management methods lack the ability to recognize sequential / non-sequential access and data access patterns, leading to low prefetch accuracy and easily causing cache pollution or invalid prefetching, which in turn negatively impacts performance. Summary of the Invention

[0005] Therefore, it is necessary to provide a data reading method, apparatus, device, and storage medium that can improve data reading efficiency in response to the above-mentioned technical problems.

[0006] Firstly, a data reading method is provided, including: Set up a cache layer within the memory; Upon receiving the first data read request, the target data information is read at the hard disk layer; Based on the physical location of the target data information in the hard disk layer and the preloading algorithm, select the data block adjacent to the location of the target data information from the hard disk layer and preload it into the cache layer; In response to receiving a second data read request, the target data information is read from the cache layer. If the target data information is not present in the cache layer, the target data information is read from the disk layer.

[0007] Secondly, a data reading device is provided, applied to a data reading method described in the first aspect, comprising: The cache settings module is used to set up a cache layer within the memory; The first data reading module is used to read target data information from the hard disk layer in response to receiving a first data read request; The data preloading module is used to select data blocks adjacent to the target data information from the hard disk layer according to the physical location of the target data information in the hard disk layer and the preloading algorithm, and preload them into the cache layer. The second data reading module is used to read the target data information from the cache layer in response to receiving a second data read request, and to read the target data information from the hard disk layer if the target data information is not present in the cache layer.

[0008] Thirdly, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the data reading method described in the first aspect.

[0009] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the data reading method described in the first aspect is implemented.

[0010] By implementing the aforementioned data reading method, apparatus, device, and storage medium, this method significantly improves data reading speed and system response efficiency by setting a cache layer within the memory and combining it with a data access pattern prediction mechanism. Upon receiving a data read request for the first time, the scheme intelligently judges access characteristics based on the physical location of the target data on the hard disk layer and historical access behavior, preloading data blocks adjacent to the target data location into the cache layer. This effectively reduces the physical I / O overhead of subsequent data accesses. When receiving a second read request, the system prioritizes reading the target data from the cache. If the cache is hit, repeated access to the hard disk is avoided, thereby accelerating data acquisition, reducing disk resource consumption, and improving overall system throughput and concurrent processing capabilities. Even if the cache is not present, the system can still fall back to hard disk access, ensuring the integrity and continuity of data access. By introducing the synergistic cooperation of the caching mechanism and the preloading strategy, not only is data read performance optimized, but the system's adaptability to sequential and non-sequential access scenarios is also enhanced, improving the intelligence and efficiency of data read operations. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a data reading method provided in an embodiment of this application; Figure 2 A structural block diagram of a data reading device provided in an embodiment of this application; Figure 3 A timing diagram of a data reading method provided in an embodiment of this application; Figure 4 This is a diagram showing the internal structure of a computer device in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] In one embodiment, such as Figure 1 As shown, a data reading method is provided, including: S100: Set up a cache layer in the memory; S200: In response to receiving the first data read request, the target data information is read at the hard disk layer; S300: Based on the physical location of the target data information in the hard disk layer and the preloading algorithm, select the data block adjacent to the location of the target data information from the hard disk layer and preload it into the cache layer; S400: In response to receiving a second data read request, the target data information is read from the cache layer. If the target data information is not present in the cache layer, the target data information is read from the disk layer.

[0017] The cache layer refers to the storage area located between memory and hard disk, used to temporarily store recently or frequently accessed data blocks to improve data access speed; the first data read request and the second data read request refer to two different read operation requests initiated by the system, which read the same or adjacent data blocks; the preloading algorithm is an algorithm that loads potentially accessed data blocks into the cache in advance based on data access patterns and historical behavior analysis; physical location refers to the actual storage location of data on the hard disk, identified by parameters such as sector number and cylinder number, used to locate the data; a hit means that the data requested by the system already exists in the cache and there is no need to access the hard disk.

[0018] Specifically, by setting up a caching layer in the storage system, when the system receives the first data read request, it not only reads the required target data but also further analyzes the physical location of that data on the hard drive. Using a preloading algorithm, the system determines which adjacent data blocks the target data might have access dependencies on, and reads these adjacent data blocks from the hard drive into the cache, thus implementing the preloading operation. When the system subsequently issues a second data read request, it first queries the cache layer. If the requested data already exists in the cache (a cache hit), the target data can be read directly from the cache, significantly reducing access latency and avoiding physical I / O operations with the hard drive. This not only improves the speed of read operations but also reduces the burden on the hard drive, extends its lifespan, and improves the overall concurrency and throughput of the system. Even if the cache is not present, the system can still fall back to read from the hard drive, ensuring the integrity and continuity of data reading. This design, combining a caching mechanism with a preloading strategy, is particularly suitable for application scenarios with certain data access patterns (such as sequential reading and big data analysis). It not only enhances the intelligence and response speed of data access, but also effectively adapts to complex and ever-changing data request patterns, enabling flexible support for different access types. It significantly optimizes the performance and response efficiency of the storage system.

[0019] In one embodiment, the method further includes: In response to receiving a second data read request and enabling the caching layer, the target data information is read from the caching layer; In response to receiving a second data read request and closing the cache layer, the target data information is read from the disk layer.

[0020] Specifically, when the system detects that the caching layer is enabled, it prioritizes reading the target data from the cache. This path depends on whether the previously pre-loaded data hits the cache. If it does, it can significantly reduce the frequency of disk access, lower data read latency, and improve overall system response speed. When the system detects that the caching layer is disabled, it means that the caching mechanism is not enabled in the current scenario, for reasons such as saving resources, handling temporary requests, or system policy adjustments. In this case, the system no longer checks the cache content but directly accesses the disk to read the target data. This ensures that even when the cache is unavailable, the system can still accurately obtain the required data, guaranteeing the continuity and integrity of data read operations and preventing data loss or access failures due to changes in cache status.

[0021] In one embodiment, in response to receiving a first data read request, target data information is read from the disk layer, which includes: a first disk layer and a second disk layer, comprising: Based on the first data read request, query the target data information at the first hard disk layer; If the target data information exists in the first hard disk layer, then the target data information is returned; If the target data information is not found in the first hard disk layer, the target data information is queried in the second hard disk layer and returned.

[0022] The first hard drive layer refers to storage media with faster access speed and shorter response time, such as high-performance SSDs or NVMe solid-state drives, used to store hot data or frequently accessed data; the second hard drive layer refers to storage devices with relatively slower access speed, larger capacity, but lower cost, such as HDDs or cold data storage media, used to store cold data or infrequently accessed data.

[0023] Specifically, the system divides the hard disk layer into a two-tier structure: a first hard disk layer and a second hard disk layer, to achieve hierarchical management of data with different access frequencies. When the system receives a first data read request, it first searches for the target data information in the first hard disk layer. Since this layer uses high-speed storage media, if the data exists, it can be returned quickly, significantly improving response speed, reducing data access latency, and lowering system I / O load. If the requested data is not found in the first hard disk layer, the system automatically switches to the second hard disk layer to continue searching and return the data. Although the access speed of the second hard disk layer is relatively slower, it typically has a larger capacity and can store a complete data system, thus ensuring the integrity and coverage of data access. This allows the system to have good data capacity and scalability while ensuring performance. This two-tier hard disk architecture requires no manual intervention from the user during implementation. The system automatically completes data search and path switching based on data popularity and storage strategy, improving the intelligence and automation level of data access.

[0024] In one embodiment, such as Figure 3 As shown, the method also includes: Upon receiving a data read request, the target data information should be queried in the cache layer. If the target data information is not found in the cache layer, the target data information is read from the disk layer and the record information is written to the head of the cache layer queue. If the target data information exists in the cache layer, the record corresponding to the target data information is moved to the head of the cache layer queue; When the cache capacity of the cache layer reaches the first threshold, data information at the tail of the cache layer queue is deleted according to the least recently used algorithm.

[0025] The head and tail of the cache queue refer to the way the cache organizes data in the form of a queue. The head of the queue represents the location of the most recently accessed data, while the tail of the queue represents the location of the least recently accessed data. The Least Recently Used (LRU) algorithm is a cache replacement strategy that prioritizes the removal of the least recently accessed data to make room for new data, thereby improving the cache hit rate.

[0026] Specifically, after receiving a data read request, the system first searches for the target data in the cache layer. If the data is found, the system immediately returns the data and moves the corresponding record to the head of the cache queue, indicating that the data has been used recently and increasing its priority in the cache. This prevents frequently accessed data from being mistakenly identified as low-frequency data and cleared, optimizing cache usage efficiency. If the cache does not exist, it means the requested data is not in the cache. In this case, the system reads the target data from the disk layer and writes its corresponding record to the head of the cache queue, giving it priority. This data also becomes a candidate for potentially high-frequency access later, improving the subsequent hit rate. To prevent the cache layer from expanding indefinitely and wasting memory resources, the system sets a cache capacity threshold. When the cached data reaches this threshold, the system triggers the Least Recently Used (LRU) algorithm, automatically clearing the record corresponding to the tail of the cache queue (i.e., the least recently accessed data), making room for new data. This replacement strategy aligns with the principle of locality of access in most applications: recently accessed data is more likely to be accessed again in the future. The system enables dynamic adjustment and intelligent updating of cached content, effectively improving cache hit rate and reducing frequent hard drive access, thereby significantly improving overall data read performance.

[0027] In one embodiment, based on the physical location of the target data information in the hard disk layer and the preloading algorithm, data information adjacent to the location of the target data information is selected from the hard disk layer and preloaded into the cache layer, including: Upon receiving the first data read request, the corresponding target data information is obtained; Obtain access logs, extract historical data read request records within a preset time range from the access logs, and calculate the physical address difference between the current data read request and the previous data read request; If the physical address difference is less than the second threshold, the target data and the first preset number of adjacent data blocks are selected for preloading using a sequential access method. If the physical address difference is greater than the second threshold, a second preset number of high-frequency access data blocks near the target data are selected and preloaded using a non-sequential access method.

[0028] Among them, physical address difference refers to the difference between the physical storage locations of target data on the hard drive, represented by sector or block offset; sequential access means that data requests follow a certain order, and the access mode is usually to access adjacent data in blocks or sequentially, which is suitable for scenarios involving large amounts of continuous data access; non-sequential access means that data requests have no fixed order, and usually access data blocks at random locations, which is suitable for complex queries or irregular data reading; preset time range is a specified historical data access period, used to filter data access records within a specific time period to analyze data access patterns; high-frequency access data block refers to storage blocks or data that are accessed multiple times within a period of time, and is often used as data to be loaded into the cache first to improve the hit rate.

[0029] Specifically, upon receiving the first data read request, the system first acquires the target data and analyzes the relevant access logs. The access logs record historical data access records within a preset time range, providing information on data request patterns over a past period. The system determines whether the current request is sequential or non-sequential by calculating the difference in physical address between the target data and the previous data request. For example, if the difference in physical address between the target data and the previous access is very small (less than a preset threshold), it indicates that this request is likely sequential. In this case, the system preloads the target data and a certain number of adjacent data blocks into the cache, preloading potentially accessed data in advance. This strategy is suitable for scenarios such as video streaming and sequential file reading, significantly reducing subsequent access latency. Conversely, if the physical address difference is large, it indicates that the current access pattern is non-sequential, meaning the data access pattern does not follow a continuous order. This typically occurs during database queries or random data access. In this case, the system preloads some frequently accessed data blocks near the target data based on the access history. These data blocks are typically frequently accessed and requested data blocks. Prioritizing the loading of these data blocks improves the cache hit rate for future requests, thereby reducing direct disk access and improving system performance. This mechanism dynamically adjusts the cache preloading strategy to ensure faster data reading during sequential access and retain frequently used data during non-sequential access, enhancing the system's intelligent predictive capabilities. By analyzing access history, the system can more efficiently determine which data needs to be prioritized for caching, avoiding unnecessary resource waste. For example, when a user is viewing a player containing multiple videos, the system will prioritize loading the currently playing video and its related content, rather than immediately loading all content in the entire video library. This saves cache space and improves the user experience. In practical applications, the preloading strategy effectively improves data reading efficiency, especially in high-frequency and sequential access scenarios, reducing disk I / O burden and improving system response speed and processing capacity. Through adaptive cache management, the system flexibly responds to different access patterns, avoiding cache space waste and optimizing storage resource utilization efficiency. Therefore, this design, which combines data access pattern analysis with preloading algorithms, can significantly improve the throughput and concurrent processing capabilities of the storage system, and enhance the system's scalability.

[0030] In one embodiment, in response to a physical address difference less than a second threshold, sequential access is performed, and the target data and a first preset number of adjacent data blocks are selected for preloading, including: Obtain the physical address number of the target data information; Based on the physical address number, select the data blocks corresponding to the first preset number of consecutive physical addresses in sequence; Load consecutive data blocks sequentially into preset locations in the cache layer; In response to the cache layer capacity reaching the first threshold, data blocks at the tail of the cache queue are deleted according to the least recently used algorithm.

[0031] Among them, the physical address number refers to the specific location number of the data on the hard disk, usually represented by the block number or sector number, which is used to uniquely identify the location of the data in the physical storage medium; the preset location refers to the memory space area reserved in the cache layer for preloading data, which is used to temporarily store data blocks that may be accessed in a specified order; the first threshold refers to the upper limit value of the cache layer capacity set by the system.

[0032] Specifically, the system extracts the physical address number from the current target data information, for example, number 5001. Based on this number, the system selects a sequence of data blocks sequentially, for example, if the first preset number is 4, then data blocks numbered 5002, 5003, 5004, and 5005 are selected. These data blocks are physically adjacent to the target data and are the most likely to be accessed next. The system loads these data blocks sequentially into preset locations in the cache layer according to their numbers. This sequential loading not only preserves the continuity of data access but also establishes a fast response path for subsequent read requests. For example, in typical sequential read scenarios such as video playback or log analysis, after a user requests the current block, they are very likely to continue requesting subsequent blocks. Preloading allows these subsequent requests to directly hit the cache, thereby significantly reducing the frequency of disk I / O access and speeding up the response time. When the amount of data in the cache layer gradually increases and reaches its capacity limit, the system starts the Least Recently Used (LRU) algorithm, prioritizing the deletion of the tail of the cache queue, i.e., the least recently accessed data blocks, to free up space to accommodate newly loaded data. Retaining data that may be used again in the near future further improves the cache hit rate.

[0033] Sequential access is loaded using the preloading formula: Load amount = k * M (where k is the current load count). The preload amount for sequential access is also limited by cache space, so the actual load amount is: Actual load amount = min(k * M, remaining cache space). When sequential access is detected (physical address difference is less than a preset threshold), the first preload is: Load amount = M (i.e., k = 1); the second preload is: Load amount = 2M (k = 2); the third preload is: Load amount = 3M (k = 3). Each loaded data block is a consecutive physical address block starting from the target data, and the number is the currently calculated actual load amount. During loading, the remaining cache space is checked. If the remaining space is insufficient, the actual remaining space is used to load (and the eviction mechanism is triggered). Let the initial parameters be: M = 100 (i.e., 100 data blocks), the total cache capacity K = 500 data blocks, and the preset threshold be 10 (i.e., if the physical address difference between two adjacent I / Os is less than 10, it is considered a sequential access). The scenario is as follows: three consecutive sequential access requests are received, and each request is processed very quickly (satisfying the condition T' < ΔT, thus triggering the k*M mode). First request (target address P0): determined as sequential access (because it is the first request, there is no preceding address, but according to the access mode, it can be set as the first request, which triggers sequential preloading) k=1, calculate the load size = 1*100=100, the remaining cache space = 500-0=500; the actual load size = min(100, 500)=100; load 100 data blocks from address P0 to P99 into the cache layer; The second request (target address P100, the difference between it and the previous address P0 is 100; assuming the check is the address difference of consecutive requests, such as the previous access ending address being P99, and this request being P100, the difference is 1 < 10, so it is determined to be a sequential access): k=2, calculated load = 2 * 100 = 200; remaining cache space = 500 - 100 = 400; Actual load size = min(200, 400) = 200; Load 200 data blocks from addresses P100 to P299 into the cache layer. The third request (target address P300, address difference 1 < 10, sequential access): k=3, calculated load size = 3 * 100 = 300; remaining cache space = 500 - (100 + 200) = 200; actual load size = min(300, 200) = 200 (because the remaining space is only 200); load 200 data blocks from address P300 to P499 into the cache layer. At this time, the cache is full (100 + 200 + 200 = 500). When sequential access preloading is triggered again, some data needs to be evicted first (according to LRU) before loading.

[0034] In one embodiment, in response to a physical address difference greater than a second threshold, indicating non-sequential access, a second preset number of high-frequency access data blocks near the target data are selected for preloading, including: The physical address of the target data information is used as the central address to determine the preset range of the physical address of the target data information; At least one data block with an access frequency higher than a preset frequency threshold is selected from the preset range and used as a candidate high-frequency access data block set. Select a second preset number of high-frequency access data blocks from the candidate high-frequency access data block set; Load frequently accessed data blocks into the preset location in the cache layer; In response to the cache layer capacity reaching a first threshold, data blocks at the tail of the cache queue are deleted according to the least recently used algorithm.

[0035] Here, the center address refers to the physical address number where the current target data information is located, serving as the midpoint for defining a range and delineating the data block interval to be analyzed; the preset range refers to the physical address interval formed by extending forward and backward from the center address of the target data, usually containing several consecutive data blocks, used for statistical analysis and filtering of frequently accessed data; the preset frequency threshold refers to the minimum access frequency standard used to determine whether a data block is a frequently accessed data block; and the second threshold refers to the maximum value of the preset difference between the preset physical addresses in the system.

[0036] Specifically, the physical address of the current target data is used as the center address, and a certain range is extended forward and backward to form a preset range (e.g., ±100 data blocks) as the analysis object. Within this range, combined with access logs or real-time statistics, all data blocks with access frequencies higher than a preset frequency threshold are selected, forming a candidate high-frequency access data block set. For example, if there are 20 data blocks within this range, and 8 of them have been accessed more than 5 times in the past 10 minutes, these 8 constitute the candidate set. The system sorts the candidate set according to access frequency or according to other strategies, and selects a second preset number of data blocks (e.g., a maximum of 5), loading them into a preset location in the cache layer. Since these data blocks have been frequently accessed historically, they are likely to be accessed again, and pre-caching can significantly reduce disk access and improve subsequent response speed. When the cache layer reaches its capacity limit, the system will again use the Least Recently Used (LRU) algorithm to delete the least recently accessed data block from the end of the cache, making room for newly loaded high-frequency data. This ensures that the data stored in the cache is data that is likely to be accessed recently, continuously optimizing the caching effect. The system not only adapts to access scenarios with strong randomness and unclear patterns, but also dynamically selects valuable data for caching by introducing statistical optimization methods.

[0037] When the difference between the physical address of the target data and the historical access address exceeds a preset threshold, a physical range is defined centered on the target address. Data blocks within this range with access frequencies higher than a dynamic threshold are selected to generate a candidate set. For each data block in the candidate set, an access popularity weight factor Hj and the baseline frequency access count j are calculated, and the preload amount is calculated using the following formula: ; in And M0=M, where M is the initial fixed amount of data loaded, N is the cache access ratio of the data amount M, where N∈(0,1], and T is the baseline access time for the amount of M data. Let N be the actual access time for the data volume, where The mapping function outputs the time T′ to reach the access ratio N, where k is the upper limit of the cache capacity, k is the load count counter, and k = (1, 2, 3...), and ΔT is the emergency threshold of the base time T.

[0038] Assuming the target data is located at physical address 0xA000, the system detects that the difference between this address and the previously accessed address 0x9F00 exceeds a preset threshold, classifying it as random access. Within a 200-address radius centered on 0xA000, four high-frequency access data blocks are detected: address 0xA050 (accessed 12 times); address 0xA120 (accessed 9 times); address 0xA0F0 (accessed 7 times); and address 0xA1A0 (accessed 4 times, below the dynamic threshold of 5 times, excluded). The base frequency is set to 5 times / minute, and the calculated access frequency weighting factors are as follows: H A050 =12 / 5=2.4; H A120 =9 / 5=1.8; H A0F0 =7 / 5=1.4; Initial parameters: basic loading unit M = 15 data blocks, baseline access time T = 80ms, emergency threshold coefficient ΔT = 0.15, load count counter k = 3. Through the mapping function... The actual access time was measured to be T′=50ms.

[0039] Calculate the time threshold: T·ΔT = 80 × 0.15 = 12 ms; Determine that T′ is within the interval 12ms ≤ 50ms ≤ 80ms; One data block; The cache size is set to K = 300 data blocks, and the currently used space ∑ = 210: Actual load constraints: M k =min(21,3,300-210)=min(21,3,90)=3; Select the three data blocks with the highest weights in the candidate set: 0xA050(H=2.4)→0xA120(H=1.8)→0xA0F0(H=1.4); After loading, the space occupied is 210+3=213; the warning level is 0.7×300=210, triggering the elimination mechanism; then the number of tail data blocks deleted is 0.2×300=60.

[0040] In one embodiment, a cache layer is provided within the memory, including: In response to high system load, the cache capacity of the cache layer is increased. In response to low system load, the cache capacity of the cache layer is reduced.

[0041] System load refers to the current resource usage of the system, including comprehensive indicators such as CPU utilization, memory utilization, disk I / O, and network traffic, which are used to measure the current operating pressure and task density of the system.

[0042] Specifically, when the system detects high system load, indicating heavy processing tasks and intensive I / O requests, frequent data reads from the hard drive will significantly impact performance. In this case, the system proactively increases the cache capacity of the cache layer, providing more space for data to reside. More frequently accessed data can be kept in the cache, reducing direct access to the hard drive, alleviating disk I / O pressure, and improving overall response speed. For example, expanding the cache space in high-concurrency scenarios allows more user requests to hit the cache, reducing latency and increasing throughput. Conversely, when the system detects low load, indicating less operational pressure and relatively abundant memory resources, there's no need to retain too much cached data to maintain performance. In this case, the system will appropriately reduce the cache capacity, freeing up some cache space for other possible tasks, improving overall resource utilization efficiency. For example, shrinking the cache during idle periods at night helps reduce memory usage and supports background tasks or energy-saving strategies.

[0043] In one embodiment, it includes: Collect current system operating status parameters; Based on the system operating status parameters, the expected data access latency of the current system is calculated using a latency prediction model. If the expected data access delay exceeds a preset delay threshold, multiple data blocks related to the target data information are read from the disk layer in advance and loaded into the cache layer. If a data read request is received and the target data information is not available in the cache layer, a fault tolerance mechanism is triggered to request the target data information from the remote data source. In response to obtaining target data information from a remote data source, the target data information is written to the cache layer and returned to the data requesting end.

[0044] The system operating status parameters include: hard disk utilization, memory usage, and network response latency. Hard disk utilization refers to the storage resource usage of the hard disk in the system, i.e., the ratio of currently used hard disk space to total space. Memory usage refers to the memory (RAM) usage in the system, i.e., the ratio of currently used memory to total memory. Network latency response refers to the time delay from sending a data request to receiving a data response, reflecting the network response speed. The latency prediction model refers to predicting possible future access latency by analyzing the current system status parameters (such as hard disk utilization, memory usage, network response, etc.) and taking measures in advance to optimize data preloading. The fault tolerance mechanism refers to the system attempting to obtain data from other data sources (such as the cloud, remote storage, etc.) when the cache is missing or hard disk read / write fails, ensuring high data availability and system stability.

[0045] Specifically, after collecting current operating status parameters such as disk usage, memory occupancy, and network response latency, the system uses a latency prediction model to calculate and estimate potential access latency. If the predicted latency exceeds a preset threshold, the system proactively reads multiple data blocks related to the target data from the disk layer in advance and preloads these data blocks into the cache layer. This is done to prepare the data before the actual latency arrives, thus smoothly transitioning to the cache read stage and minimizing the impact on user experience. When an actual data read request arrives, the system first checks if the cache layer has already hit the target data. If the cache layer does not hit the target data, the system triggers a fault tolerance mechanism, attempting to retrieve the data from a remote data source or cloud data source. This mechanism ensures that even if the target data is not in the local cache, the system can still retrieve the data from external resources in a timely manner, ensuring continuous data availability and rapid response. Once the target data is successfully retrieved from the remote data source, the system stores the data in the cache layer and returns it to the requesting client. This not only improves data availability but also reduces the pressure on disk access and network requests, achieving more efficient data management and a better user experience.

[0046] In one embodiment, such as Figure 2 As shown, a data reading device is provided, including: a cache setting module 510, a first data reading module 520, a data preloading module 530, and a second data reading module 540, used for: The cache setting module 510 is used to set up a cache layer within the memory; The first data reading module 520 is used to read target data information at the hard disk layer in response to receiving a first data read request; The data preloading module 530 is used to select data blocks adjacent to the location of the target data information from the hard disk layer according to the physical location of the target data information in the hard disk layer and the preloading algorithm, and preload them into the cache layer. The second data reading module 540 is used to read target data information from the cache layer in response to receiving a second data read request, and to read target data information from the hard disk layer if the target data information does not exist in the cache layer.

[0047] In one embodiment, the first data reading module 520 is used for: Based on the first data read request, query the target data at the first hard disk layer; If the target data information exists in the first hard disk layer, then the target data information is returned; If the target data information does not exist in the first hard disk layer, the target data information is queried in the second hard disk layer and returned.

[0048] It should be understood that, although Figure 2 The steps in the device block diagram are shown sequentially as indicated by the arrows; however, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be executed in other orders. Furthermore, Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0049] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data reading method embodiments when it is run.

[0050] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0051] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data reading method embodiments.

[0052] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, such as Figure 4 As shown, to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0053] The data reading method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data reading method, characterized in that, include: Set up a cache layer within the memory; Upon receiving the first data read request, the target data information is read from the hard disk layer; Based on the physical location of the target data information in the hard disk layer and the preloading algorithm, a data block adjacent to the location of the target data information is selected from the hard disk layer and preloaded into the cache layer; In response to receiving a second data read request, the target data information is read from the cache layer; if the target data information is not present in the cache layer, the target data information is read from the hard disk layer.

2. The data reading method according to claim 1, characterized in that, The method further includes: In response to receiving the second data read request and enabling the cache layer, the target data information is read from the cache layer; In response to receiving the second data read request and closing the cache layer, the target data information is read from the hard disk layer.

3. The data reading method according to claim 1, characterized in that, In response to receiving a first data read request, target data information is read from the hard disk layer, which includes: a first hard disk layer and a second hard disk layer, comprising: Based on the first data read request, query the target data information in the first hard disk layer; If the target data information exists in the first hard disk layer, then the target data information is returned; If the target data information is not found in the first hard disk layer, the target data information is queried in the second hard disk layer and returned.

4. The data reading method according to claim 1, characterized in that, The method further includes: In response to receiving a data read request, the target data information is queried in the cache layer; If the target data information is not found in the cache layer, the target data information is read from the disk layer and the record information is written to the head of the cache layer queue. In response to the presence of the target data information in the cache layer, the record corresponding to the target data information is moved to the head of the cache layer queue; In response to the cache capacity of the cache layer reaching a first threshold, data information at the tail of the cache layer queue is deleted according to the least recently used algorithm.

5. The data reading method according to claim 1, characterized in that, Based on the physical location of the target data information in the hard disk layer and the preloading algorithm, a data block adjacent to the location of the target data information is selected from the hard disk layer and preloaded into the cache layer, including: In response to receiving the first data read request, the corresponding target data information is obtained; Obtain access logs, extract historical data read request records within a preset time range from the access logs, and calculate the physical address difference between the current data read request and the previous data read request; In response to the physical address difference being less than the second threshold, the target data and the first preset number of adjacent data blocks are selected for preloading using a sequential access method; If the physical address difference is greater than the second threshold, a second preset number of high-frequency access data blocks near the target data are selected for preloading using a non-sequential access method.

6. The data reading method according to claim 5, characterized in that, In response to the physical address difference being less than the second threshold, sequential access is performed, and the target data and a first preset number of adjacent data blocks are selected for preloading, including: Obtain the physical address number of the target data information; Based on the physical address number, select data blocks corresponding to a first preset number of consecutive physical addresses in sequence; The consecutive data blocks are loaded sequentially into the preset positions of the cache layer; In response to the cache layer capacity reaching a first threshold, the data block at the tail of the cache queue is deleted according to the least recently used algorithm.

7. The data reading method according to claim 5, characterized in that, In response to the physical address difference being greater than the second threshold, indicating non-sequential access, a second preset number of high-frequency access data blocks near the target data are selected for preloading, including: Using the physical address of the target data information as the central address, a preset range of the physical address of the target data information is determined; At least one data block with an access frequency higher than a preset frequency threshold is selected from the preset range and used as a candidate high-frequency access data block set. A second preset number of the high-frequency access data blocks are selected from the candidate high-frequency access data block set; Load the high-frequency access data block into a preset position in the cache layer; In response to the cache layer capacity reaching the first threshold, data blocks at the tail of the cache queue are deleted according to the least recently used algorithm.

8. A data reading device, characterized in that, The device includes: The cache settings module is used to set up a cache layer within the memory; The first data reading module is used to read target data information from the hard disk layer in response to receiving a first data read request; The data preloading module is used to select data blocks adjacent to the location of the target data information from the hard disk layer according to the physical location of the target data information in the hard disk layer and the preloading algorithm, and preload them into the cache layer; The second data reading module is used to read the target data information from the cache layer in response to receiving a second data read request, and to read the target data information from the hard disk layer if the target data information does not exist in the cache layer.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images