A skip-reading pre-reading method and device based on distributed file system

Abstract

The present invention provides a skip read and pre-read method and device based on a distributed file system, and relates to the field of computer technology. Aiming at the skip read performance bottleneck existing in the traditional file system, the present invention records and analyzes the data reading offset and length, calculates the step length of the file skip read, and realizes multi-mode recognition, including strict skip read, 2-element strict alternating skip read and local random skip read. On the basis of recognition, an intelligent pre-reading mechanism is introduced, and the pre-reading strategy is optimized according to the skip read mode, which significantly improves the data reading performance of the system. In addition, the present invention also has the advantages of low cost and high compatibility, and is suitable for fields such as large-scale data processing and distributed computing, and can effectively improve the file data access efficiency and provide a better user experience. The present invention can effectively identify different skip read modes and optimize the skip read and pre-read operations accordingly to improve the system data reading performance.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file systems, and in particular to a skip read pre-reading method and device based on a distributed file system. Background Art

[0002] With the continuous development of computer applications, file systems play a vital role in various fields for storing and managing various types of data. In large-scale data processing and distributed computing scenarios, file reading performance is particularly important. However, due to the diversity of file storage methods and access modes, there are many performance bottlenecks, one of which is the file skipping operation.

[0003] File skipping refers to reading data in a file not in order, but skipping some data blocks according to certain rules. This skipping reading method can improve reading efficiency in some scenarios, but it also brings some problems. Traditional file systems have some limitations in supporting skipping operations. For example, they can only support skipping with a fixed step length, which makes it impossible to meet the requirements of variable step length in actual applications. In addition, the traditional pre-reading mechanism also fails to give full play to the advantages of skipping operations, resulting in a decrease in system performance when processing large-scale skipping.

[0004] Therefore, in the current distributed file system, the current file skipping and pre-reading mechanisms have some problems, mainly including:

[0005] (1) Problem of constant step size: Traditional file systems only support skipping with fixed step sizes, which cannot meet the requirements of variable step sizes in practical applications, resulting in performance degradation.

[0006] (2) Insufficient pre-reading mechanism: The traditional pre-reading mechanism fails to effectively adapt to skip reading operations, resulting in limited pre-reading effect when the system handles skip reading.

[0007] (3) Skip mode recognition problem: The current file system fails to effectively recognize and optimize different skip modes and cannot provide targeted optimization measures for different modes.

[0008] Therefore, a new method and device are needed to identify skip read patterns of different step lengths and optimize the pre-reading mechanism accordingly to improve the data reading performance of the distributed file system. Summary of the invention

[0009] In view of this, the present invention proposes a skip reading and pre-reading method and device based on a distributed file system, which realizes more efficient file skip reading and pre-reading and improves system performance by recording and analyzing data reading offset and length, performing pattern recognition, penalty mechanism and pre-reading mechanism; the present invention can effectively identify different skip reading modes and optimize skip reading and pre-reading operations accordingly to improve system data reading performance.

[0010] Based on the above purpose, in a first aspect, the present invention provides a skip read-ahead method based on a distributed file system, the method comprising the following steps:

[0011] Record and analyze the data read offset (off) and data read length (len), and calculate the file skipping step length (skip);

[0012] Perform pattern recognition and determine the type of skipping pattern according to the change of skipping step length, including strict skipping, two-element strict alternating skipping and local random skipping;

[0013] Implement a penalty mechanism to handle cache hits and misses to maintain the continuity of the skip read pattern;

[0014] According to different skipping modes, corresponding pre-reading mechanisms are designed, including strict skipping pre-reading, 2-ary strict alternating skipping pre-reading, and arithmetic difference skipping and local random skipping pre-reading.

[0015] As a further solution of the present invention, the skip mode is strict skip, and the strict skip pre-reading mechanism includes:

[0016] Read the current data block for the first time;

[0017] Pre-read two data blocks: (off+len+skip,len) and (off+2*len+2*skip,len);

[0018] A pre-read mark is set at 1 / 2 of the pre-read data length, and the number of pre-read data blocks is gradually increased. The total length of the pre-read data blocks has an upper limit;

[0019] The pre-reading request is sent in an asynchronous cycle.

[0020] As a further solution of the present invention, the skip mode is 2-element strict alternating skip, and the 2-element strict alternating skip pre-reading mechanism includes:

[0021] Pre-read at least two data blocks each time, including (off+len+skip,len) and (off+2*len+skip1+skip2,len);

[0022] After a cache hit, gradually increase the number of pre-read data blocks;

[0023] The pre-read mark position and the upper limit of the pre-read length are the same as those of strict skip.

[0024] As a further solution of the present invention, the skipping mode is arithmetic skipping and local random skipping, and the arithmetic skipping and local random skipping pre-reading mechanisms include:

[0025] Calculate the pre-read length according to the accumulated value of the skip step length;

[0026] Perform local random pre-reading, and set the pre-reading mark at 1 / 3 of the pre-reading length;

[0027] When data is read to the previous position of the arithmetic jump read, it is determined whether the cache exists, and if not, data is pre-read.

[0028] As a further solution of the present invention, pattern recognition comprises the following steps:

[0029] If skip_list[].size=1, it is determined to be in strict skip mode;

[0030] If skip_list[].size=2, and skip_i=skip_i+2, it is determined to be a 2-element strict alternating skip mode;

[0031] If skip_list[].size>50, it is determined to be in random skip mode;

[0032] Perform an arithmetic sequence verification on each skip_i[]. If it is an arithmetic sequence and the number of skip_list[].size skip_i N < 10, mark it as a local random read, otherwise mark it as a random skip read.

[0033] As a further solution of the present invention, the penalty mechanism includes the following situations:

[0034] If the cache hits, continue to use the identified pattern;

[0035] If the cache misses, no penalty will be imposed if the cache data that misses is in the pre-read data;

[0036] If the cache misses, and if the miss data has not been pre-read, pattern recognition is performed again, and the original cache data is not cleared temporarily.

[0037] As a further solution of the present invention, during the pattern recognition process, the skip calculation results for each group of data records are recorded in skip_list[] in the form of (skip_i, skip_i[]), where skip_i[] represents the order of the same skip during the reading process for pattern recognition.

[0038] In a second aspect, the present invention further provides a skip read pre-reading device based on a distributed file system, comprising:

[0039] The data analysis module is used to record and analyze the offset (off) and data reading length (len) of the data read, and calculate the step length (skip) of the file skipping;

[0040] A pattern recognition module is used to determine the type of skipping pattern according to the change of the skipping step length, including strict skipping, two-element strict alternating skipping and local random skipping;

[0041] The penalty mechanism module is used to handle cache hits and misses to maintain the continuity of the skip mode;

[0042] The pre-reading mechanism module is used to design corresponding pre-reading mechanisms for different skip reading modes, including strict skip reading pre-reading, 2-element strict alternating skip reading pre-reading, and arithmetic difference skip reading and local random skip reading pre-reading.

[0043] As a further solution of the present invention, the skip mode is strict skip, and the strict skip pre-reading mechanism includes:

[0044] Pre-read two data blocks: (off+len+skip,len) and (off+2*len+2*skip,len);

[0045] A pre-read mark is set at 1 / 2 of the pre-read data length, and the number of pre-read data blocks is gradually increased. The total length of the pre-read data blocks has an upper limit;

[0046] The pre-reading request is sent in an asynchronous cycle.

[0047] As a further solution of the present invention, the skip mode is 2-element strict alternating skip, and the 2-element strict alternating skip pre-reading mechanism includes:

[0048] Pre-read at least two data blocks each time, including (off+len+skip,len) and (off+2*len+skip1+skip2,len);

[0049] After a cache hit, gradually increase the number of pre-read data blocks;

[0050] The pre-read mark position and the upper limit of the pre-read length are the same as those of strict skip.

[0051] As a further solution of the present invention, the skipping mode is arithmetic skipping and local random skipping, and the arithmetic skipping and local random skipping pre-reading mechanisms include:

[0052] Calculate the pre-read length according to the accumulated value of the skip step length;

[0053] Perform local random pre-reading, and set the pre-reading mark at 1 / 3 of the pre-reading length;

[0054] When data is read to the previous position of the arithmetic jump read, it is determined whether the cache exists, and if not, data is pre-read.

[0055] As a further solution of the present invention, the pattern recognition module comprises the following steps:

[0056] If skip_list[].size=1, it is determined to be in strict skip mode;

[0057] If skip_list[].size=2, and skip_i=skip_i+2, it is determined to be a 2-element strict alternating skip mode;

[0058] If skip_list[].size>50, it is determined to be in random skip mode;

[0059] Perform an arithmetic sequence verification on each skip_i[]. If it is an arithmetic sequence and the number of skip_list[].size skip_i N < 10, mark it as a local random read, otherwise mark it as a random skip read.

[0060] As a further solution of the present invention, the penalty mechanism module includes the following situations:

[0061] If the cache hits, continue to use the identified pattern;

[0062] If the cache misses, no penalty will be imposed if the cache data that misses is in the pre-read data;

[0063] If the cache misses, and if the miss data has not been pre-read, pattern recognition is performed again, and the original cache data is not cleared temporarily.

[0064] As a further solution of the present invention, the data analysis module records the offset (off) and data reading length (len) of 101 groups of data, and calculates the step length (skip) of file skipping to form a data basis for pattern recognition.

[0065] As a further solution of the present invention, the device can identify different skip reading modes and optimize skip reading and pre-reading operations accordingly, thereby effectively improving the data reading performance of the system distributed file system.

[0066] In another aspect of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, any one of the skip-reading pre-reading methods based on a distributed file system according to the present invention is executed.

[0067] In yet another aspect of the present invention, a computer-readable storage medium is provided, storing computer program instructions, which, when executed, implement any of the skip-reading and pre-reading methods based on a distributed file system according to the present invention.

[0068] Compared with the prior art, the skip read pre-reading method and device based on the distributed file system proposed in the present invention has the following beneficial effects:

[0069] 1. Optimize data reading performance: The present invention adopts optimization strategies for different skip reading modes, effectively identifies and optimizes different modes such as strict skip reading, 2-element strict alternating skip reading and local random skip reading, thereby optimizing data reading in different scenarios and significantly improving the data reading performance of the distributed file system.

[0070] 2. Enhanced pre-reading mechanism: The pre-reading mechanism of the traditional file system fails to fully adapt to the skip read operation, while the present invention designs a targeted pre-reading strategy according to the actual skip read mode, making the pre-reading more intelligent and efficient. Through a reasonable pre-reading mechanism, the data cache can be fully utilized in the skip read operation, the number of IO operations can be reduced, and the data access speed can be further accelerated.

[0071] 3. Flexible adaptation to skip reading changes: The present invention can adapt to changes in different skip reading patterns in real time through pattern recognition and penalty mechanisms; even if the skip reading pattern changes, it can quickly re-identify and adjust the pre-reading strategy, thereby maintaining efficient data reading performance and providing a better user experience.

[0072] 4. Improve system bandwidth performance: The pre-reading mechanism and optimization strategy of the present invention can effectively improve the bandwidth performance of file skipping operations, especially in large-scale data processing and distributed computing scenarios. By reducing unnecessary data reading delays and IO operations, system resources can be more fully utilized and the overall data reading efficiency can be improved.

[0073] 5. Low cost and high compatibility: The skip read pre-reading method and device based on the distributed file system of the present invention is compatible with the existing distributed storage engine, without large-scale changes to the existing system structure. At the same time, the design concept of the present invention is simple, the additional resource consumption is low, and significant performance improvement can be achieved without increasing too much system overhead.

[0074] In summary, the skip-reading pre-reading method and device based on a distributed file system of the present invention have significant beneficial effects in improving data reading performance, optimizing the pre-reading mechanism, and adapting to skip-reading changes, and is expected to be widely used in large-scale data processing, distributed computing and other fields.

[0075] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0077] In the figure:

[0078] Figure 1 The present invention is a flowchart of a method for skipping and pre-reading based on a distributed file system according to an embodiment of the present invention.

[0079] Figure 2 The present invention is a flowchart of a strict skip read-ahead mechanism in a skip read-ahead method based on a distributed file system in an embodiment of the present invention.

[0080] Figure 3 The present invention is a flowchart of a two-element strict alternating skip read in a skip read-ahead method based on a distributed file system according to an embodiment of the present invention.

[0081] Figure 4 The present invention is a flowchart of a medium-difference skip read and a local random skip read pre-read mechanism in a skip read pre-read method based on a distributed file system in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0083] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0084] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, device, product or equipment that includes a series of steps or units.

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0086] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0087] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0088] Since the current file system includes support for file skipping in the Linux system, it only supports the situation where the jump step size remains unchanged and the data reading length remains unchanged. This scenario is relatively special and has a narrow application scope in practice. In current application scenarios, the step size in skipping is variable, but the data reading length generally does not change.

[0089] In order to cope with this variable step length skipping, the present invention designs a skip reading and pre-reading method based on the file system. The present invention can identify skip readings of different step lengths and support the pre-reading mechanism of the corresponding mode, thereby increasing the bandwidth of skip reading data and improving the system data reading performance.

[0090] See also Figure 1 As shown, an embodiment of the present invention provides a skip read pre-reading method based on a distributed file system, the method comprising the following steps:

[0091] Step S10, record and analyze the offset (off) and data reading length (len) of the data read, and calculate the step length (skip) of the file skipping;

[0092] Step S20, performing pattern recognition, and determining the type of skipping pattern according to the change of the skipping step length, including strict skipping, two-element strict alternating skipping and local random skipping;

[0093] Step S30: Implement a penalty mechanism to handle cache hits and misses to maintain the continuity of the skip mode;

[0094] Step S40: Design corresponding pre-reading mechanisms for different skip reading modes, including strict skip reading pre-reading, 2-ary strict alternating skip reading pre-reading, and arithmetic difference skip reading and local random skip reading pre-reading.

[0095] In this embodiment, see Figure 2 As shown, the skip mode is strict skip, and the strict skip pre-reading mechanism includes:

[0096] Step S101, reading the current data block for the first time;

[0097] Step S102, pre-read two data blocks (off+len+skip,len) and (off+2*len+2*skip,len);

[0098] Step S103, setting a pre-read mark at 1 / 2 of the pre-read data length, gradually increasing the number of pre-read data blocks, and the total length of the pre-read data blocks has an upper limit;

[0099] Step S104: Use an asynchronous cyclic sending method to make a pre-read request.

[0100] In this embodiment, see Figure 3 As shown, the skip mode is 2-element strict alternating skip, and the 2-element strict alternating skip pre-reading mechanism includes:

[0101] Step S201, pre-read at least two data blocks each time, including (off+len+skip,len) and (off+2*len+skip1+skip2,len);

[0102] Step S202: after a cache hit, gradually increase the number of pre-read data blocks;

[0103] Step S203: The pre-reading mark position and the upper limit of the pre-reading length are the same as those of the strict skipping.

[0104] In this embodiment, see Figure 4 As shown, the skipping mode is arithmetic skipping and local random skipping, and the arithmetic skipping and local random skipping pre-reading mechanisms include:

[0105] Step S301, calculating the pre-read length according to the accumulated value of the skip step length;

[0106] Step S302, perform local random pre-reading, and set the pre-reading mark at 1 / 3 of the pre-reading length;

[0107] Step S303: when the data is read to the previous position of the arithmetic jump, it is determined whether the cache exists, and if not, data pre-reading is performed.

[0108] In this embodiment, pattern recognition includes the following steps:

[0109] If skip_list[].size=1, it is determined to be in strict skip mode;

[0110] If skip_list[].size=2, and skip_i=skip_i+2, it is determined to be a 2-element strict alternating skip mode;

[0111] If skip_list[].size>50, it is determined to be in random skip mode;

[0112] Perform an arithmetic sequence verification on each skip_i[]. If it is an arithmetic sequence and the number of skip_list[].size skip_i N < 10, mark it as a local random read, otherwise mark it as a random skip read.

[0113] In this embodiment, the penalty mechanism includes the following situations:

[0114] If the cache hits, continue to use the identified pattern;

[0115] If the cache misses, no penalty will be imposed if the cache data that misses is in the pre-read data;

[0116] If the cache misses, and if the miss data has not been pre-read, pattern recognition is performed again, and the original cache data is not cleared temporarily.

[0117] In this embodiment, during the pattern recognition process, the skip calculation results for each group of data records are recorded in skip_list[] in the form of (skip_i, skip_i[]), where skip_i[] represents the order of the same skip during the reading process for pattern recognition.

[0118] Exemplarily, the offset off and the data reading length len of 101 sets of data reading are recorded and analyzed. The data analysis includes:

[0119] last_end_off=last_off+len;

[0120] skip=off-last_end_off;

[0121] The calculated skip is recorded in skip_list[]. The elements in the linked list are as follows:

[0122] (skip_i, skip_i[]): skip represents the jump step length, skip_i[] records the order of the same skip in the reading process, and the form of skip_i[] is as follows:

[0123] skip_i[]={ki,kj,…kn}

[0124] Among them: the subscript of K is the order in which the data IO is read.

[0125] Among them, pattern recognition includes:

[0126] skip_list[].size=1 means that skip will remain unchanged, that is, strict skip reading;

[0127] skip_list[].size=2,skip_i=skip_i+2, which means 2-element strictly alternating skipping

[0128] skip_list[].size>50, that is, random skipping of skip changes;

[0129] Perform an arithmetic sequence verification on each skip_i[]. If it is an arithmetic sequence, it means that the step length of skip_i is skipped in an arithmetic order; skip_list[].size–number of skip_i = N

[0130] If N<10, it is marked as local random read; otherwise, it is marked as random skip read.

[0131] The penalty mechanism includes:

[0132] After the data model is determined, subsequent data reading needs to be verified and judged;

[0133] If the cache hits, the identified pattern will continue to be used;

[0134] In case of cache miss, if the cache data that misses is being pre-read, no penalty will be imposed; if the cache data that misses is not being pre-read, pattern recognition needs to be performed again, and the original cache data will not be cleared for the time being.

[0135] The pre-reading mechanism includes a strict skip pre-reading mechanism, a two-element strict alternating skip pre-reading mechanism, an arithmetic difference skip pre-reading mechanism, and a local random skip pre-reading mechanism; the strict skip pre-reading mechanism includes:

[0136] The first actual read of the current data block is to pre-read two data blocks, (off+len+skip,len) and (off+2*len+2*skip,len). After a hit, the number of pre-read data blocks is doubled, and the pre-read mark is placed at 1 / 2 of the pre-read data length. The total length of the pre-read data blocks has an upper limit;

[0137] Pre-read requests are sent in an asynchronous cycle (in a queued form).

[0138] Among them, the two-element strictly alternate skipping includes:

[0139] At least 2 data blocks (off+len+skip,len) and (off+2*len+skip1+skip2,len) are pre-read each time. After a cache hit, the number of pre-read data blocks is doubled, and the pre-read mark position and pre-read length upper limit are the same as those of strict skip read.

[0140] Among them, arithmetic skip reading and local random skip reading and pre-reading include:

[0141] By cumulatively adding up the corresponding (skip+len) values, we can get the pre-read length of L. After determining it as a local random read, each pre-read is L, and the pre-read mark is 1 / 3L. When the data is read to the previous position of the arithmetic skip read, it is necessary to determine whether the cache exists. If not, data is pre-read.

[0142] In this embodiment, when creating ghost cache record analysis data, it includes:

[0143] Create a data analysis container to store 101 skipped data, and use the LRU method to eliminate data;

[0144] Create a pattern recognition recorder to store the recognized skip patterns, and its records need to be written to disk. After the pattern is created, it is written to disk at the first cache hit.

[0145] Create the required temporary cache linked list and skip read pre-read parameter structure in the system to facilitate the analysis of recorded data and storage of skip read parameters.

[0146] The specific implementation steps are as follows:

[0147] A. Data analysis: Record 101 sets of data and calculate the skip step length of the file:

[0148] last_end_off=last_off+len;

[0149] skip=off-last_end_off;

[0150] The calculated skip is recorded in skip_list[]. The elements in the linked list are as follows:

[0151] (skip_i, skip_i[]): skip represents the jump step length, skip_i[] records the order of the same skip in the reading process, and the form of skip_i[] is as follows:

[0152] skip_i[]={ki,kj,…kn}

[0153] Among them: the subscript of K is the order of data IO in the reading process;

[0154] B. Pattern Recognition:

[0155] skip_list[].size=1 means that skip will remain unchanged, that is, strict skip reading;

[0156] skip_list[].size=2,skip_i=skip_i+2, which means 2-element strictly alternating skipping

[0157] skip_list[].size>50, that is, random skipping of skip changes;

[0158] Perform an arithmetic sequence verification on each skip_i[]. If it is an arithmetic sequence, it means that the step length of skip_i is skipped in an arithmetic order; skip_list[].size–number of skip_i = N

[0159] N<10, marked as local random read; otherwise marked as random skip read;

[0160] C. Penalty mechanism:

[0161] After the data model is determined, subsequent data reading needs to be verified and judged;

[0162] 1. If the cache hits, the identified pattern will continue to be used;

[0163] 2. If the cache miss occurs, no penalty will be imposed if the cache data that misses is being pre-read. If the miss data is not pre-read, pattern recognition needs to be performed again, and the original cache data will not be cleared for the time being.

[0164] D. Pre-reading mechanism:

[0165] 1. Strict skip read and pre-read mechanism:

[0166] The first actual read of the current data block is to pre-read two data blocks, (off+len+skip,len) and (off+2*len+2*skip,len). After a hit, the number of pre-read data blocks is doubled, and the pre-read mark is placed at 1 / 2 of the pre-read data length. The total length of the pre-read data blocks has an upper limit;

[0167] Pre-read requests are sent asynchronously in a loop (in a queued form);

[0168] 2.2 Strict Alternation Skip Reading

[0169] At least 2 data blocks (off+len+skip,len) and (off+2*len+skip1+skip2,len) are pre-read each time. After a cache hit, the number of pre-read data blocks is doubled, and the pre-read mark position and pre-read length upper limit are the same as those of strict skip read.

[0170] 3. Arithmetic skipping and local random skipping pre-reading:

[0171] The corresponding (skip+len) values ​​are cumulatively added together to obtain the pre-read length L. After determining that it is a local random read, each pre-read is L, and the pre-read mark is 1 / 3L. When the data is read to the previous position of the arithmetic skip read, it is necessary to determine whether the cache exists. If not, data is pre-read.

[0172] Therefore, the present invention provides a skip read pre-reading method based on a distributed file system, aiming to optimize the file data reading performance. The skip read operation in the traditional file system often causes performance bottlenecks, especially in scenarios such as distributed computing and large-scale data processing. Specifically:

[0173] Firstly, the invention calculates the step length of file skipping by recording the offset and length of data reading according to different data reading situations. Then, through pattern recognition, multiple skipping modes are identified, including strict skipping, two-element strict alternating skipping and local random skipping, so as to accurately determine the skipping behavior.

[0174] Secondly, the present invention introduces an intelligent pre-reading mechanism, which flexibly optimizes the pre-reading strategy according to different skip reading modes. For the strict skip reading mode, a multi-level pre-reading method is adopted to gradually increase the number of pre-read data blocks; for the 2-element strict alternating skip reading mode, alternating pre-reading is implemented to improve the cache hit rate; for the local random skip reading mode, accurate data pre-reading is achieved through arithmetic sequence verification and pre-reading marking. These pre-reading mechanisms effectively reduce the number of IO operations, reduce data reading delay, and improve data access speed.

[0175] At the same time, the present invention also includes a penalty mechanism to maintain the continuity of the skip read mode. For cache hits and misses, intelligent processing is performed according to the pre-read state to ensure that the system continues to operate in the optimized read mode.

[0176] Finally, the present invention has the characteristics of low cost and high compatibility, and can be smoothly integrated into the existing distributed storage engine without large-scale changes to the existing system structure.

[0177] In summary, the skip-reading pre-reading method and device based on a distributed file system of the present invention effectively improves data reading performance through multi-mode recognition and intelligent pre-reading mechanism, is applicable to many fields such as large-scale data processing and distributed computing, and brings significant improvement to the data access efficiency of the file system in a distributed environment.

[0178] It should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0179] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0180] According to a second aspect of the embodiments of the present invention, the present invention further provides a skip read pre-reading device based on a distributed file system, comprising:

[0181] The data analysis module is used to record and analyze the offset (off) and data reading length (len) of the data read, and calculate the step length (skip) of the file skipping;

[0182] A pattern recognition module is used to determine the type of skipping pattern according to the change of the skipping step length, including strict skipping, two-element strict alternating skipping and local random skipping;

[0183] The penalty mechanism module is used to handle cache hits and misses to maintain the continuity of the skip mode;

[0184] The pre-reading mechanism module is used to design corresponding pre-reading mechanisms for different skip reading modes, including strict skip reading pre-reading, 2-element strict alternating skip reading pre-reading, and arithmetic difference skip reading and local random skip reading pre-reading.

[0185] As a further solution of the present invention, the skip mode is strict skip, and the strict skip pre-reading mechanism includes:

[0186] Pre-read two data blocks: (off+len+skip,len) and (off+2*len+2*skip,len);

[0187] A pre-read mark is set at 1 / 2 of the pre-read data length, and the number of pre-read data blocks is gradually increased. The total length of the pre-read data blocks has an upper limit;

[0188] The pre-reading request is sent in an asynchronous cycle.

[0189] In this embodiment, the skip mode is 2-element strict alternating skip, and the 2-element strict alternating skip pre-reading mechanism includes:

[0190] Pre-read at least two data blocks each time, including (off+len+skip,len) and (off+2*len+skip1+skip2,len);

[0191] After a cache hit, gradually increase the number of pre-read data blocks;

[0192] The pre-read mark position and the upper limit of the pre-read length are the same as those of strict skip.

[0193] As a further solution of the present invention, the skipping mode is arithmetic skipping and local random skipping, and the arithmetic skipping and local random skipping pre-reading mechanisms include:

[0194] Calculate the pre-read length according to the accumulated value of the skip step length;

[0195] Perform local random pre-reading, and set the pre-reading mark at 1 / 3 of the pre-reading length;

[0196] When data is read to the previous position of the arithmetic jump read, it is determined whether the cache exists, and if not, data is pre-read.

[0197] In this embodiment, the pattern recognition module includes the following steps:

[0198] If skip_list[].size=1, it is determined to be in strict skip mode;

[0199] If skip_list[].size=2, and skip_i=skip_i+2, it is determined to be a 2-element strict alternating skip mode;

[0200] If skip_list[].size>50, it is determined to be in random skip mode;

[0201] Perform an arithmetic sequence verification on each skip_i[]. If it is an arithmetic sequence and the number of skip_list[].size skip_i N < 10, mark it as a local random read, otherwise mark it as a random skip read.

[0202] In this embodiment, the penalty mechanism module includes the following situations:

[0203] If the cache hits, continue to use the identified pattern;

[0204] If the cache misses, no penalty will be imposed if the cache data that misses is in the pre-read data;

[0205] If the cache misses, and if the miss data has not been pre-read, pattern recognition is performed again, and the original cache data is not cleared temporarily.

[0206] In this embodiment, the data analysis module records the offset (off) and data reading length (len) of 101 sets of data, and calculates the step length (skip) of file skipping to form a data basis for pattern recognition.

[0207] In this embodiment, the device can identify different skip reading modes and optimize skip reading and pre-reading operations accordingly, thereby effectively improving the data reading performance of the system distributed file system.

[0208] The present invention increases the recognition of skip reading and pre-reading modes, and improves the skip reading performance; the skip reading modes that can be recognized by this design method at present are:

[0209] 1. Strictly skip reading: {a,a,a,…a,a,a}

[0210] 2.2 Strict skipping of elements: {a,b,a,b…a,b,a,b}

[0211] 3. Multivariate local random skip (less than 10 elements): {(a,b,c,d),(a,c,d,b),(a,d,c,b)…(b,a,c,d),(b,d,a,c)}

[0212] Compared with the prior art, the present invention has the following advantages:

[0213] 1. Performance. The distributed file system has many identifiable skip read modes, which can enhance the file reading bandwidth performance of the system;

[0214] 2. Low cost: The additional resource consumption generated by the present invention is maintained at a low level.

[0215] 3. Compatibility: The technology of the present invention can be integrated into the existing distributed storage engine.

[0216] The present invention provides a skip read pre-reading device based on a distributed file system, which is dedicated to solving the performance bottleneck of traditional file systems in skip read operations, as well as data reading challenges in scenarios such as distributed computing and large-scale data processing. Specifically:

[0217] First, the skip read pre-reading device includes a data analysis module, which records the offset and length of the data read, calculates the step length of the file skip read, and forms a data basis for pattern recognition. Through recording and calculation, the device can accurately obtain the characteristics of data reading.

[0218] Secondly, the skip reading and pre-reading device also includes a pattern recognition module, which can intelligently identify different skip reading modes according to the change of the skip reading step length. These modes include strict skip reading, two-element strict alternating skip reading and local random skip reading. Through pattern recognition, the device can accurately determine the skip reading behavior.

[0219] At the same time, the skip read pre-reading device has an intelligent penalty mechanism module to intelligently handle cache hits and misses. For cache hits, the recognized pattern is continuously used; for misses, intelligent judgment is made based on the pre-reading status to ensure that the system continues to run in the optimized reading mode.

[0220] Most importantly, the skip read pre-reading device introduces a pre-reading mechanism and designs corresponding pre-reading strategies according to different skip read modes. For the strict skip read mode, the pre-reading efficiency is improved by gradually increasing the number of pre-read data blocks and setting pre-reading marks; for the 2-element strict alternating skip read mode, alternating pre-reading is implemented to further improve the data access speed; for the arithmetic skip read and local random skip read modes, the pre-reading length is calculated according to the cumulative value, local random pre-reading is implemented, and data cache utilization is optimized.

[0221] In summary, the skip-reading and pre-reading device based on the distributed file system of the present invention has been optimized and innovated in many aspects, and can efficiently cope with different data reading modes, improve file data reading performance, and reduce IO latency, so that it is suitable for various scenarios such as distributed computing and large-scale data processing, and provides a significant improvement in the data access efficiency of the distributed file system.

[0222] According to a third aspect of an embodiment of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of any one of the above embodiments is implemented.

[0223] The computer device includes a processor and a memory, and may also include an input device and an output device. The processor, the memory, the input device and the output device may be connected via a bus or other means, and the input device may receive input digital or character information, and generate a signal input related to the migration of skip read-ahead based on a distributed file system. The output device may include a display device such as a display screen.

[0224] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the skip read-ahead method based on the distributed file system in the embodiment of the present application. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating device, an application required for at least one function; the data storage area may store data created by the use of the skip read-ahead method based on the distributed file system, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0225] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data. The processors of multiple computer devices of the computer device of this embodiment execute various functional applications and data processing of the server by running non-volatile software programs, instructions and modules stored in the memory, that is, implementing the steps of the skip read pre-reading method based on the distributed file system of the above method embodiment.

[0226] It should be understood that, in the absence of conflict, all the embodiments, features and advantages described above for the skip-reading pre-reading method based on a distributed file system according to the present invention are also applicable to the skip-reading pre-reading and storage medium based on a distributed file system according to the present invention.

[0227] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given of the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire device. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope disclosed in the embodiments of the present invention.

[0228] Finally, it should be noted that the computer-readable storage medium (e.g., memory) herein may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. As an example and not by way of limitation, a nonvolatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. A volatile memory may include a random access memory (RAM), which may act as an external cache memory. As an example and not by way of limitation, RAM may be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0229] The various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein may be implemented or executed using the following components designed to perform the functions herein: a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and / or any other such configuration.

[0230] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0231] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0232] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the embodiments of the present invention as above, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A skip read and pre-read method based on a distributed file system, characterized in that: The following steps are involved: Record and analyze the data read offset and data read length, and calculate the file skipping step length; Perform pattern recognition and determine the type of skipping pattern according to the change of skipping step length; If skip_list[].size = 1, it is determined to be a strict skip mode; if skip_list[].size = 2, and skip_i = skip_i+2, it is determined to be a binary strict alternating skip mode; if skip_list[].size > 50, it is determined to be an arithmetic skip mode and a local random skip mode; wherein, an arithmetic sequence is verified for each skip_i[]. If it is an arithmetic sequence and the number of skip_list[].size skip_i N < 10, it is marked as a local random read, otherwise it is marked as a random skip; wherein, skip represents the jump step length, and skip_i[] records the order of the same skip in the reading process for pattern recognition; Implement a penalty mechanism to handle cache hits and misses to maintain the continuity of the skip read pattern; the penalty mechanism includes: if the cache hits, continue to use the recognized pattern; if the cache misses, if the cache data that misses is in the pre-read data, no penalty is imposed; if the cache misses, if the data that misses is not pre-read, re-perform pattern recognition, and the original cache data is not cleared temporarily; Design corresponding pre-reading mechanisms for different skip reading modes, including strict skip reading pre-reading mechanism, 2-element strict alternating skip reading pre-reading mechanism and arithmetic difference skip reading and local random skip reading pre-reading mechanism; The strict skip read pre-read mechanism includes: reading the current data block for the first time; pre-reading two data blocks (off + len + skip, len) and (off + 2 * len + 2 * skip, len); setting a pre-read mark at 1 / 2 of the pre-read data length, gradually increasing the number of pre-read data blocks, and the total length of the pre-read data blocks has an upper limit; using an asynchronous cyclic sending method to make pre-read requests; where off represents the offset of data reading, and len represents the data reading length; The 2-element strict alternating skip read pre-reading mechanism includes: pre-reading at least two data blocks each time, including (off + len + skip, len) and (off + 2 * len + skip1 + skip2, len); after a cache hit, gradually increasing the number of pre-read data blocks; the pre-reading mark position and the pre-reading length upper limit are the same as those of the strict skip read; The arithmetic skip and local random skip pre-reading mechanism includes: calculating the pre-reading length according to the skip step accumulation value; performing local random pre-reading, and setting the pre-reading mark at 1 / 3 of the pre-reading length; when the data is read to the previous position of the arithmetic skip, judging whether the cache exists, and if not, performing data pre-reading.

2. The skip-reading and pre-reading method based on a distributed file system according to claim 1, characterized in that: During the pattern recognition process, the skip calculation results for each set of data records are recorded in skip_list[] in the form of (skip_i, skip_i[]).

3. A skip-reading pre-reading device based on a distributed file system, characterized in that: Used to execute the skip-reading and pre-reading method based on a distributed file system according to claim 1 or 2, the skip-reading and pre-reading device based on a distributed file system comprises: The data analysis module is used to record and analyze the offset and length of data read, and calculate the step length of file skipping; A pattern recognition module is used to determine the skip mode type according to the change of the skip step length; if skip_list[].size = 1, it is determined to be a strict skip mode; if skip_list[].size = 2, and skip_i = skip_i+2, it is determined to be a 2-element strict alternating skip mode; if skip_list[].size > 50, it is determined to be an arithmetic skip mode and a local random skip mode; wherein, an arithmetic sequence is verified for each skip_i[]. If it is an arithmetic sequence and the number N of skip_list[].size skip_i < 10, it is marked as a local random read, otherwise it is marked as a random skip; wherein, skip represents the jump step length, and skip_i[] records the order of the same skip in the reading process for pattern recognition; The penalty mechanism module is used to process cache hits and misses to maintain the continuity of the skip read mode; the penalty mechanism includes: if the cache hits, the recognized pattern is continuously used; if the cache misses, if the cache data that misses is in the pre-read data, no penalty is performed; if the cache misses, if the miss data is not pre-read, the pattern recognition is performed again, and the original cache data is not cleared temporarily; A pre-reading mechanism module is used to design corresponding pre-reading mechanisms for different skip reading modes, wherein the pre-reading mechanisms include a strict skip reading pre-reading mechanism, a two-element strict alternating skip reading pre-reading mechanism, and an arithmetic skip reading and local random skip reading pre-reading mechanism; wherein the strict skip reading pre-reading mechanism includes: reading the current data block for the first time; pre-reading two data blocks (off + len + skip, len) and (off + 2 * len + 2 * skip, len); setting a pre-reading mark at 1 / 2 of the pre-read data length, gradually increasing the number of pre-read data blocks, and the total length of the pre-read data blocks has an upper limit; using an asynchronous cyclic sending method to make a pre-reading request; wherein off represents an offset for data reading, and len represents a data reading length; the two-element strict alternating skip reading pre-reading mechanism includes: pre-reading at least two data blocks each time, including (off + len + skip, len) and (off + 2 * len + skip1 + skip2,len); after a cache hit, gradually increase the number of pre-read data blocks; the pre-read mark position and the upper limit of the pre-read length are the same as those of the strict skip read; the arithmetic skip read and local random skip read pre-reading mechanism include: calculating the pre-reading length according to the skip read step cumulative value; performing local random pre-reading, and setting the pre-reading mark at 1 / 3 of the pre-reading length; when the data is read to the previous position of the arithmetic skip read, determine whether the cache exists, if not, perform data pre-reading.

Images