Rapid data retrieval method and system

Abstract

The invention relates to the technical field of data retrieval, in particular to a rapid data retrieval method and system, and the method comprises the following steps: based on a received data retrieval query request, positioning to a target data block, obtaining an access type weight and an access frequency count, and generating a data block heat value; according to the method, after a data retrieval query request is received, a target data block is quickly positioned by analyzing query content, related information of an access type and access frequency is synchronously extracted, a static access behavior is converted into a quantifiable heat value index, and the old and new degree of the data block is extracted by introducing a time difference calculation mode; therefore, the heat value is not a static counting result any more, but is a dynamic feature expression with an aging judgment basis. And a dynamic popularity evaluation value is further generated through combination of the static popularity and the aging factor, so that comprehensive judgment on the current importance and activeness of the data block is realized, and data value sorting logic is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data retrieval, in particular to a data fast retrieval method and system. BACKGROUND

[0002] The technical field of data retrieval relates to efficient positioning and extraction of target information in massive data, and is widely used in search engines, database systems, content management platforms, big data analysis engines and other systems.

[0003] The existing technology has problems such as significant response delay, passive data scheduling, and rigid resource allocation in the face of data retrieval tasks in high concurrency and large data scenarios. In the current commonly used data retrieval system, the positioning of target data often depends on static index structure and fixed strategy path, ignoring the frequency change and real-time hotness distribution of data access, which makes it difficult to effectively distinguish and schedule hot data and cold data. At the same time, the existing technology is difficult to determine whether some data has current access value, and only uses past total access frequency as the basis for evaluation, which easily leads to unreasonable occupation of cache space. Therefore, improvement is needed. SUMMARY

[0004] The purpose of the present application is to solve the shortcomings in the prior art and provide a data fast retrieval method and system.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme, a data fast retrieval method comprising the following steps: Based on the received data retrieval query request, the target data block is located and the access type weight and access frequency count are obtained, and the data block hotness value is generated; Based on the data block hotness value, the global clock is called to obtain the time difference, the timeliness hotness weight is obtained, the timeliness hotness weight is operated based on the data block hotness value, and the dynamic hotness evaluation value is obtained; Based on the dynamic hotness evaluation value, the migration threshold of the storage level where the data block is located is compared and judged, the level migration decision signal is generated, the target storage level of the data block is matched and determined based on the level migration decision signal, and the data block migration instruction is established; Based on the data block migration instruction, the source and destination addresses of data movement are constructed, the target migration path is obtained, the data block is prefetched by the background daemon process based on the target migration path, and the data retrieval optimization layout is established.

[0006] Preferably, the data block hotness value obtaining step is: Based on the received data retrieval query request, the data block identification field attached in the query request is parsed, the data block identification field is matched to the global storage mapping table, the access record entry set of the corresponding data block is retrieved, and the access type field sequence and the access timestamp field sequence are formed; According to the access type field sequence and the access timestamp field sequence, the access times in the set time window are classified and counted according to the access type, the time sequence set of each type of access is constructed, the change rate is calculated according to the corresponding timestamp, and the access type count mapping table and the access type change rate index table are obtained. According to the access type count mapping table and the access type change rate index table, the data block hotness value is calculated.

[0007] Preferably, the time-sensitive hotness weight obtaining step is: Based on the data block hotness value, the hotness value generation timestamp contained in the generated record is retrieved, and the standard timestamp of the current time is synchronously obtained. The difference in seconds between the current timestamp and the hotness value generation timestamp is calculated to form the hotness value time difference result. According to the hotness value time difference result, the pre-defined time difference reference interval table is matched, and each time difference value and the upper and lower limit ranges of the interval are contrasted in turn to determine the time decay section number where the hotness value time difference result is located, and the section identification information corresponding to the number is extracted to generate the time decay section identification. Based on the time decay section identification, the numerical parameter content of the corresponding section is read from the set time-sensitive decay factor parameter table as the time-sensitive influence factor corresponding to the current data block to generate the time-sensitive hotness weight.

[0008] Preferably, the dynamic hotness evaluation value obtaining step is: Based on the time-sensitive hotness weight, the hotness value corresponding to each data block is extracted to obtain the data block hotness value and the time-sensitive hotness weight. According to the data block hotness value and the time-sensitive hotness weight, a dynamic hotness input vector group is constructed, all data block indexes are extracted, the data block position corresponding to each group of hotness parameters is labeled, and the hotness saturation threshold and the time-sensitive influence factor are appended as constant items to form a hotness calculation input set. Based on the hotness calculation input set, the dynamic hotness evaluation value is calculated.

[0009] Preferably, the hierarchical migration decision signal obtaining step is: Based on the dynamic hotness evaluation value, the migration threshold value of the storage hierarchy where the data block is located is extracted, the hierarchical type identification corresponding to the migration threshold value is parsed piece by piece, and the comparison-verified dynamic hotness evaluation value and the migration threshold value contrast set are generated. According to the dynamic hotness evaluation value after the comparability check and the migration threshold value control set, the numerical difference of each data block dynamic hotness evaluation value and the corresponding storage level migration threshold value is calculated one by one, the difference result between the dynamic hotness evaluation value and the migration threshold value is recorded, and the positive and negative states of the difference result are marked respectively to generate the difference mark set of the dynamic hotness evaluation value and the migration threshold value; Based on the difference mark set of the dynamic hotness evaluation value and the migration threshold value, the data block mark state with the numerical difference greater than or equal to zero in the difference mark set is extracted, the difference mark state is taken as the migration judgment basis, the upshift migration action of the storage level is triggered when the mark state is positive, the downshift migration action of the storage level is triggered when the mark state is negative, and the direction and category of the migration action are recorded to generate the level migration decision signal.

[0010] Preferably, the data block migration instruction obtaining step is: Based on the level migration decision signal, the migration action direction and category identification carried in the level migration decision signal are parsed piece by piece, the target storage level information corresponding to the migration action direction identification is extracted, and the data block target storage level identification is generated; According to the data block target storage level identification, the storage unit address and port number corresponding to the target storage level identification are matched, the capacity state and current available space value of the target storage unit are obtained, it is judged whether the available space value of the target storage unit meets the size requirement of the current data block, and the data block target storage unit effectiveness confirmation result is formed; Based on the data block target storage unit effectiveness confirmation result, for the target storage unit whose available space value meets the size requirement of the data block, the data block migration instruction content is combined and constructed, including the unique identifier of the data block, the source storage unit position of the data block, the target storage unit address and port number, and the data block migration priority information and transmission mode mark are attached, and the data block migration instruction is generated.

[0011] Preferably, the target migration path obtaining step is: Based on the data block migration instruction, the storage system management interface is called to obtain the source storage unit address of the data block and the destination storage unit address after migration, and the source and destination address set of data movement is formed; According to the source and destination address set of data movement, the path query of the source storage unit address and the destination storage unit address is carried out, the network nodes and link information associated in the path are extracted, the data transmission link node sequence from the source storage unit address to the destination storage unit address is determined, and the target migration path is formed.

[0012] Preferably, the data retrieval optimization layout obtaining step is: Based on the target migration path, a background daemon is called to sequentially traverse the link node sequence recorded in the target migration path, to perform data block prefetching on each node, to gradually cache the data blocks to the intermediate nodes in the order of the data transmission link node sequence, to update the cache layout index table to indicate the new layout position of the data blocks, and to establish a data retrieval optimization layout.

[0013] The application also provides a data fast retrieval system, comprising: A query identification module, based on the received data retrieval query request, locates to the target data block and obtains the access type weight and the access frequency count to generate a data block heat value; A heat evaluation module, based on the data block heat value, calls a global clock to obtain a time difference, obtains a timeliness heat weight, and performs operation based on the timeliness heat weight and the data block heat value to obtain a dynamic heat evaluation value; A hierarchical decision module, based on the dynamic heat evaluation value, compares and judges the migration threshold of the storage hierarchy where the data block is located, generates a hierarchical migration decision signal, matches and determines the target storage hierarchy of the data block based on the hierarchical migration decision signal, and establishes a data block migration instruction; A data prefetching module, based on the data block migration instruction, constructs the source and destination addresses of data movement, obtains a target migration path, and performs data block prefetching by a background daemon based on the target migration path to establish a data retrieval optimization layout.

[0014] Compared with the prior art, the application has the advantages and positive effects that: In the application, after receiving a data retrieval query request, the target data block is quickly located by analyzing the query content, and the access type and access frequency related information are extracted synchronously, the static access behavior is converted into a quantifiable heat value index, the time difference calculation method is introduced to extract the new and old degree of the data block, so that the heat value is no longer a static counting result, but a dynamic characteristic expression with timeliness judgment basis. Further, the dynamic heat evaluation value is generated by combining the static heat and the timeliness factor to realize comprehensive judgment of the current importance and activity of the data block, and the data value sorting logic is strengthened. On this basis, the dynamic heat evaluation value is compared with the migration threshold of the storage hierarchy to construct a decision signal to complete the migration trigger judgment between different storage hierarchies, and the linkage mechanism of heat and storage strategy is opened. Through the clear target storage hierarchy positioning of the data block, the source and destination address path is constructed, the background daemon is introduced for prefetching before migration execution, the data on the migration path is preloaded and layout adjustment operation is combined, so that the subsequent data retrieval process has higher hit rate and response speed. In summary, the application improves retrieval efficiency, reduces access delay, optimizes storage load allocation, and provides sustained support for high-concurrency retrieval requirements in large-scale data environment. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The schematic diagram of the steps of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0017] Please refer to Figure 1 The present application provides a technical scheme, a data fast retrieval method, comprising the following steps: Based on the received data retrieval query request, the target data block is located and the access type weight and the access frequency count are obtained, and the data block hotness value is generated; Based on the data block hotness value, the time difference is obtained by calling the global clock, the timeliness hotness weight is obtained, the timeliness hotness weight is operated based on the data block hotness value, and the dynamic hotness evaluation value is obtained; Based on the dynamic hotness evaluation value, the migration threshold of the storage level where the data block is located is compared and judged, the level migration decision signal is generated, the target storage level of the data block is matched and determined based on the level migration decision signal, and the data block migration instruction is established; Based on the data block migration instruction, the source and destination addresses of data movement are constructed, the target migration path is obtained, the data block is prefetched by the background daemon process based on the target migration path, and the data retrieval optimization layout is established.

[0018] The data block hotness value acquisition step is: Based on the received data retrieval query request, the data block identification field attached in the query request is parsed, the data block identification field is matched to the global storage mapping table, the access record entry set of the corresponding data block is retrieved, the access type field sequence and the access timestamp field sequence are formed; According to the access type field sequence and the access timestamp field sequence, the access times in the set time window are classified and counted according to the access type, the time sequence set of each type of access is constructed, and the change rate is calculated according to the corresponding timestamp, the access type count mapping table and the access type change rate index table are obtained; According to the access type count mapping table and the access type change rate index table, the data block hotness value is calculated, and the calculation formula is: ; Wherein, is the data block hotness value, is the total number of access types, is the current access type serial number, is the first a type factor of class access, the first total number of class access records, the current timestamp, the first timestamp of the first access in the class access, a time decay constant, the first relative change rate of the class access within a time window, a change rate adjustment coefficient.

[0019] Specifically, based on the received data retrieval query request, the system first calls a query parser, which performs lexical analysis and syntax analysis on the request string, such as a structured query language (SQL) statement SELECT data FROM table WHERE block_id='7A3F9B', according to predefined syntax rules, extracts the data block identifier field '7A3F9B' as the key index, then uses this identifier field as the key value to perform a quick search in the global storage mapping table in the resident memory, which adopts a hash table structure, the key is the data block identifier, and the value is a composite structure containing a physical location pointer and an access log index, for example {'7A3F9B': {'location':'ssd_array_02 / volume_3 / offset_8192', 'log_index': 'access_log_2023_q4:10245'}}, the physical location pointer can directly locate the data block, and the access log index points to the log file recording all historical access operations of the data block and the starting line number, the system then opens the corresponding log file according to the index, and sequentially scans from the specified line number, loads each access record related to '7A3F9B' such as {'timestamp': 1696154400, 'access_type':'read', 'user': 'app_server_01'} into the temporary buffer area, finally, the system traverses all record entries in the buffer area, extracts the access_type field and timestamp field respectively, organizes them into two independent time-ordered sequences, forming the access type field sequence and the access timestamp field sequence.

[0020] According to the access type field sequence and the access timestamp field sequence obtained in the previous step, the system sets a dynamically adjusted time window, the size of which is determined according to the current load and data access mode of the system, for example, 15 minutes during peak transaction time and extended to 60 minutes during off-peak time. The system takes the current time as the base point, and traces back all access records within this time window, and groups the records according to the access type, such as'read', 'write', 'update', and then accurately counts the number of records in each group, thereby constructing an access type count mapping table with the structure {'read': 210, 'write': 35, 'update': 12}. At the same time, in order to calculate the access change rate, the system further divides the set time window into two consecutive and equal length subintervals, namely the recent subinterval and the prior subinterval, for example, a 60-minute time window is divided into the last 0-30 minutes and the previous 31-60 minutes. The system respectively counts the number of occurrences of each access type in the two subintervals, denoted as Count_recent and Count_prior, and applies the formula (Count_recent-Count_prior) / (Count_prior+1) to calculate the relative change rate, where the denominator plus 1 is to avoid the error of dividing by zero. Finally, the change rate results of various types of access are stored in the access type change rate index table, such as {'read': 0.15, 'write': -0.2, 'update': 0.5}, obtaining the access type count mapping table and the access type change rate index table.

[0021] Formula: The formula has the benefit of comprehensively evaluating the three core dimensions of data block hotness through a unified framework: the inherent cost of access behavior, the temporal contiguity of access behavior, and the future trend of access behavior. Specifically, by introducing the parameter, it distinguishes the consumption difference of different access types (such as read, write) on system resources; through the exponential decay term , it ensures that the more recent access contributes more to the hotness value, which conforms to the principle of locality of data access; through the trend adjustment term , it makes the hotness evaluation have foresight, can amplify the hotness of data blocks with rising access frequency, and vice versa, thereby realizing dynamic, accurate and predictive quantitative evaluation of data hotness, providing high-quality input basis for subsequent storage layering decisions.

[0022] is a constant predefined in the system, which represents all kinds of data access operations that the system can identify and process, and this value is determined when the system architecture is designed, and the calculation of all data blocks remains consistent. For example, a typical database system may define four access types as read, write, update, and delete, so the value of is 4.

[0023] is the type factor of the th access type, which quantifies the inherent impact of different types of access operations on system resources and business importance. Its setting is not subjective, but based on the analysis of historical performance data of the system. The specific steps are as follows: first, continuously monitor and record the average CPU cycles consumed by each type of access operation (such as read, write) during execution ( ), the number of disk I / O operations ( ), and the occupied network bandwidth ( ), then, normalize the collected resource consumption data to the [0, 1] interval, the normalization formula is: , finally, according to the business requirements, assign weights to different resource types ( , , , and the sum is 1), and calculate the final type factor of each access type by weighted summation, the calculation formula is: For example, after monitoring, the resource consumption of write operation is generally higher than that of read operation, after setting the weight , the type factor of read operation ( ) is , and the type factor of write operation ( ) is .

[0024] is the total number of the th access record, which directly comes from the access type count mapping table generated in the previous step, representing the total frequency of a specific type of data access within a specified time window. This parameter is a direct reflection of the recent activity level of the data block. For example, from the access type count mapping table {'read': 210, 'write': 35}, we can know that for read operation ( ), the total number of accesses is , and for write operation ( ), the total number of accesses is .

[0025] The current timestamp is obtained by the system in real time by calling the global clock service each time the heat calculation is performed. It is usually expressed as the total number of seconds or milliseconds since a certain fixed point in time (such as the Unix epoch). It provides a unified and dynamic time base for heat calculation, ensuring the accuracy of the time decay effect calculation. For example, in a calculation, the current timestamp obtained by the system is 1696158000.

[0026] For the first The first in class access The timestamp of each access is extracted from the access timestamp field sequence generated in the previous steps. It records the time when each specific access operation occurred and is the basic data for calculating access proximateness. For example, for a read operation of a data block, its three most recent access timestamps are 1696157995, 1696157980, and 1696157950, respectively.

[0027] As a time decay constant, this parameter controls the rate at which the influence of historical access records decays over time. Its value is set based on the half-life of data access, i.e., the probability and timeframe for a data block to be accessed again after it has been accessed. By analyzing the historical access intervals of a large amount of data, statistically analyzing their probability distribution, and fitting an exponential decay model, a suitable value is determined. The goal is to maintain the impact of data access frequency at a high level within its typical reuse cycle, while decreasing rapidly outside of that cycle. For example, if analysis shows that the average reuse cycle of data is 30 minutes (1800 seconds), it is expected that its frequency contribution will decay to its original value at this point. Then it can be made Calculations yielded .

[0028] For the first The relative rate of change of access type within the time window, which comes directly from the access type change rate index table generated in the previous steps, reflects the short-term trend of data block access frequency. It is a key indicator for assessing whether the data block's popularity is in an upward or downward phase. For example, from the access type change rate index table {'read': 0.15, 'write': -0.2}, we can see the relative rate of change of read operations. The relative rate of change of write operations .

[0029] This is the rate of change adjustment coefficient. This parameter is used to adjust the sensitivity of the heat calculation to the trend of changes in access frequency. Its value is calibrated through multiple rounds of simulation tests in a pre-production environment, where different... The value is selected to achieve the best balance between improving the hit rate of hot data and avoiding invalid migration of cold data, and the accuracy of migration decision and the performance of data retrieval of the observation system are observed to improve the effect For example, it is found through testing that when the value is set to 0.8, the overall performance of the system is optimal, and therefore the value is determined to be 0.8 . .

[0030] Calculation process: In the last 60-minute time window, the access situation is as follows: Read operation ( ): ,The access timestamp is 10 seconds and 600 seconds away from the current time , respectively, and the relative change rate .

[0031] Write operation ( ): ,The access timestamp is 300 seconds away from the current time , and the relative change rate .

[0032] System parameters: ,. .

[0033] Calculate the hotness value contributed by the read operation ( ): Calculate the sum of the time decay term: ; Calculate the trend adjustment term: ; Calculate : ; Calculate the hotness value contributed by the write operation ( ): Calculate the sum of the time decay term: ; Calculate the trend adjustment term: ; Calculate : ; Calculate the final data block hotness value : ; ; The result shows that the comprehensive hotness value of the data block is 3.5778, which is a quantitative score combining access type, access timeliness and access trend, and this value will be used as the basis for subsequent dynamic hotness evaluation, for comparison with the migration threshold of the storage hierarchy. If the value is higher than the upward migration threshold of the current hierarchy, it indicates that the data block is becoming more and more active and needs to be migrated to a higher performance storage medium; otherwise, if it is lower than the downward migration threshold, it indicates that its access value is decreasing and should be migrated to a lower cost storage hierarchy.

[0034] The acquisition step of the timeliness hotness weight is: Based on the data block hotness value, the hotness value generation timestamp contained in the generation record is retrieved, and the standard timestamp of the current time is synchronously acquired, the difference in seconds between the current timestamp and the hotness value generation timestamp is calculated, and the hotness value time difference result is formed; According to the hotness value time difference result, the pre-defined time difference reference interval table is matched, and each time difference value is compared with the upper and lower limit ranges of the interval in turn to determine the time decay section number where the hotness value time difference result is located, and the section identification information corresponding to the number is extracted to generate the time decay section identification. Based on the time decay section identification, the numerical parameter content of the corresponding section is read from the set timeliness decay factor parameter table as the timeliness influence factor corresponding to the current data block, and the timeliness hotness weight is generated.

[0035] Specifically, based on the data block hotness value record obtained in the last calculation period, for example, {'block_id': '7A3F9B', 'hotness': 3.5778, 'timestamp': 1696158000}, the system first accurately extracts the hotness value generation timestamp from the record, i.e. 1696158000, which is a standard Unix timestamp representing the exact time when the hotness value calculation is completed. Then, the system initiates a query request to the network time protocol (NTP) synchronized local system clock service to obtain the standard timestamp of the current time, for example, 1696161600. Subsequently, the system performs a subtraction operation, i.e. 1696161600-1696158000, to calculate the total number of seconds between the two timestamps, and obtains a difference of 3600 seconds. This difference is encapsulated as the hotness value time difference result.

[0036] According to the heat value time difference result, that is, 3600 seconds, the system will be matched with a predefined time difference reference interval table next, which is established by statistical analysis of the system historical data block life cycle, and the specific establishment process is: analyze the time interval distribution of the data block from becoming a hotspot for the first time to the access frequency significantly decreasing in the past few months, and divide these time intervals into four representative categories using the K-Means clustering algorithm. The centroid of each category defines the typical time effectiveness of the section, thereby determining the boundaries of the interval. For example, the table structure is {(section number, interval lower limit (seconds), interval upper limit (seconds), section identification information),...}, and the specific entries are {(1, 0, 600, 'extremely high time effectiveness'), (2, 601, 3600, 'high time effectiveness'), (3, 3601, 86400,'medium time effectiveness'), (4, 86401, 604800, 'low time effectiveness')}. The system will start an iterative matching process to compare the heat value time difference result 3600 seconds with each interval in the table in turn. First, 3600 is compared with the first section [0, 600], which does not match. Then, 3600 is compared with the second section [601, 3600], which falls exactly on the closed upper limit of the interval, and is determined to be a successful match. The system immediately stops iteration, locks the section number of the matched section as 2, and extracts the corresponding section identification information 'high time effectiveness' from the entry to generate the time decay section identification.

[0037] Based on the time decay section identification, that is, 'high time effectiveness', the system looks up the corresponding value from a static configured time effectiveness decay factor parameter table. The role of this parameter table is to map the qualitative time effectiveness description to the quantitative weight factor, and its content is set based on a series of simulation experiments: in a sandbox isolated from the production environment, for different storage levels and data access modes, run multiple groups of data migration strategies, each group of strategies uses different decay factor combinations, and by evaluating the average data retrieval delay and storage cost of each group of strategies within a certain time, select the group of decay factor values that maximizes the comprehensive benefit function benefit = (α * retrieval performance improvement) - (β * storage cost increase) as the final configuration, where α and β are weight coefficients set according to business priorities, for example, α = 0.7, β = 0.3. The final obtained parameters are represented as follows: {'extremely high time effectiveness': 1.0, 'high time effectiveness': 0.85,'medium time effectiveness': 0.5, 'low time effectiveness': 0.2}. The system uses 'high time effectiveness' as the key to query in this parameter table and directly obtains the corresponding value 0.85. This value is determined as the time effectiveness influence factor corresponding to the current data block and is directly output as the final result to generate the time effectiveness heat weight.

[0038] The acquisition steps of the dynamic heat evaluation value are: Based on the timeliness hotness weight, the hotness value corresponding to each data block is extracted to obtain the data block hotness value and the timeliness hotness weight; According to the data block hotness value and the timeliness hotness weight, a dynamic hotness input vector group is constructed, all data block indexes are extracted, the data block position corresponding to each group of hotness parameters is labeled, and a hotness saturation threshold and a timeliness influence factor are appended as constant terms to form a hotness calculation input set; Based on the hotness calculation input set, a dynamic hotness evaluation value is calculated, and the calculation formula is:

[0039] Specifically, based on the timeliness hotness weight generated by the system for each data block, for example, the timeliness hotness weight corresponding to the data block '7A3F9B' is 0.85, the system starts a data aggregation process, which first traverses the list of all data block identifiers that need to be evaluated for hotness, for each data block identifier in the list, the system takes it as a key value and queries the data block hotness cache area in the memory, which is a key value storage structure storing records such as {'block_id': '7A3F9B', 'hotness': 3.5778, 'timestamp': 1696158000}, the system extracts the data block hotness value, i.e., 3.5778, from it, then the system pairs the data block hotness value just extracted with the timeliness hotness weight corresponding to the data block to form a temporary data structure containing the data block identifier, the data block hotness value and the timeliness hotness weight, for example, {'block_id': '7A3F9B', 'Q': 3.5778, 'Z': 0.85}, this operation is performed for all data blocks to be evaluated, and finally a set containing all data block hotness values and timeliness hotness weights is formed.

[0040] ​​​​​​​​​According to the data block heat value and the timeliness heat weight set obtained in the previous step, the system first organizes this set into a structured dynamic heat input vector group, where each vector represents a data block and contains its identifier, heat value Q and timeliness heat weight Z. Then, the system reads two system-level parameters as constant items from the global configuration, the first being the heat saturation threshold, which is set to prevent excessive influence on the evaluation results caused by unlimited growth of the heat value. The value is determined by the system's daily timed analysis of the historical distribution of all data block heat values in the past 30 days, and the calculation of its 98th percentile. The percentile value is used as the heat saturation threshold. For example, after calculation, the 98th percentile of the historical heat value is 50.0, and the heat saturation threshold is set to 50.0. The second is the timeliness impact factor, which is used to adjust the contribution of timeliness to the final evaluation value. The value is determined by simulation testing in an offline environment. In the test, historical access data is used to traverse a series of candidate impact factor values (e.g. from 0.1 to 1.0, with a step of 0.1), calculate the simulated cache hit rate of the data migration strategy under different factors, and select the factor value that can achieve the highest hit rate. For example, the test results show that when the factor is 0.5, the system performance is best, so the timeliness impact factor is set to 0.5. Finally, the system adds these two constant items to the dynamic heat input vector group to form the heat calculation input set.

[0041] Formula: The advantage of the formula is that it introduces a dynamic, nonlinear adjustment factor based on the original data block heat value , which realizes fine-tuned correction of heat evaluation. Specifically, the exponential term can amplify the original heat according to the timeliness heat weight , and this amplification effect is adjusted by the timeliness impact factor . More importantly, the term in the denominator introduces a saturation suppression mechanism. When the original heat of a data block is already very high and close to the saturation threshold , the denominator will become larger, thereby suppressing the additional growth caused by timeliness. This design avoids over-amplification of hot data, while allowing timeliness to have a more significant impact on data blocks with moderate heat or those that are "pre-heating", thereby identifying data with immediate access value.

[0042] Data block hotness value, representing the inherent access activity of a data block within a specific time window, derived from the hotness calculation formula in the previous step, is the base input of this formula, which comprehensively reflects access frequency, access type, and recent access history. In this example, the result calculated for data block '7A3F9B' in the previous step is used, with a data block hotness value of 3.5778.

[0043] Time-sensitive hotness weight, indicating the freshness of the data block hotness value, is derived from the difference between the hotness value generation time and the current time in the previous step, with a value range of 0 to 1. The higher the value, the more "fresh" the hotness value, reflecting the current access pattern. In this example, the result calculated for data block '7A3F9B' in the previous step is used, with a time-sensitive hotness weight of 0.85.

[0044] Time-sensitive influence factor, a global adjustment coefficient, is used to control the time-sensitive hotness weight , affecting the strength of the final evaluation value. This parameter is optimized by backtesting historical data. The specific steps are as follows: collect all data block access logs and corresponding migration decision records of the system in the past three months, set an evaluation function such as predicting the cache hit rate after migration, then test a series of values (for example, from 0.1 to 2.0, with a step of 0.1) in a parameter optimization framework (such as grid search), recalculate the dynamic hotness evaluation value of historical data for each value and simulate migration decisions, and finally select the value that maximizes the evaluation function value (i.e., the highest cache hit rate) as the system parameter. For example, after the above process, the optimal time-sensitive influence factor is determined to be 0.5.

[0045] Hotness saturation threshold, a global constant, is used to define the "soft upper limit" of the hotness value, preventing highly hot data blocks from being too far apart from other data blocks after time-sensitive adjustment. This threshold is set based on statistical analysis of long-term running data of the system. The specific steps are as follows: the system automatically calculates all data block hotness values in the past 30 days every day, forming a large data set, then calculates the probability density distribution of the data set, and selects a specific percentile point at the tail of the distribution as the threshold, usually choosing the 98th percentile or the 99th percentile to ensure that only a few extreme hotspots will reach the saturation zone. For example, the analysis result shows that the 98th percentile of historical hotness value is 50.0, so the hotness saturation threshold is set to 50.0.

[0046] Calculation process: According to the heat calculation input set, the parameters of the data block '7A3F9B' are extracted: Data block heat value .

[0047] Time-sensitive heat weight .

[0048] Time-sensitive influence factor .

[0049] Heat saturation threshold .

[0050] Substitute the above parameters into the formula for calculation: Calculate the heat saturation influence term in the denominator: ; Calculate the exponential part: ; Calculate the exponential term, i.e. the time-sensitive adjustment factor: ; Calculate the final dynamic heat evaluation value : ; The results show that after time-sensitive weighting and saturation inhibition adjustment, the final dynamic heat evaluation value of data block '7A3F9B' is 5.3195, which is about 48.7% higher than the original heat value 3.5778, indicating that the data block not only has certain basic heat, but also its heat information is very "fresh" and has high potential for immediate access.

[0051] The acquisition step of the hierarchical migration decision signal is: Based on the dynamic heat evaluation value, extract the migration threshold value of the storage hierarchy where the data block is located, parse the hierarchical type identifier corresponding to the migration threshold value by piece, generate the dynamic heat evaluation value and the migration threshold value comparison set after comparability verification; According to the dynamic heat evaluation value and the migration threshold value comparison set after comparability verification, calculate the numerical difference value of the dynamic heat evaluation value of each data block and the corresponding storage hierarchy migration threshold value one by one, record the difference value result between the dynamic heat evaluation value and the migration threshold value, mark the positive and negative states of the difference value result respectively, and generate the difference value mark set of the dynamic heat evaluation value and the migration threshold value; Based on the difference value mark set of the dynamic heat evaluation value and the migration threshold value, extract the data block mark state with the numerical difference value greater than or equal to zero in the difference value mark set, take the difference value mark state as the migration judgment basis, trigger the storage hierarchy up migration action when the mark state is positive, trigger the storage hierarchy down migration action when the mark state is negative, and record the direction and category of the migration action, and generate the hierarchical migration decision signal.

[0052] Specifically, based on the dynamic hotness evaluation value of each data block calculated in the previous step, for example, the dynamic hotness evaluation value of data block '7A3F9B' is 5.3195, the system first determines the current storage level of the data block by querying the global storage mapping table, for example, it is determined to be located in "HDD_Tier" layer, then the system accesses a tier migration threshold table pre-configured by the system administrator according to the performance indicators (such as IOPS, bandwidth, delay) and cost model of each storage medium, which defines in detail the hotness conditions required for each storage level to migrate upwards or downwards, and its basis is: long-term monitoring of the average read-write delay of each level under different loads, the delay control target (for example, the delay of SSD level is less than 1 millisecond, and the delay of HDD level is less than 15 milliseconds) is converted into the corresponding dynamic hotness evaluation value threshold through historical data regression analysis of performance and hotness, for example, for "HDD_Tier" level, the threshold for migrating upwards to "SSD_Tier" is set to 5.0, and the threshold for migrating downwards to "Archive_Tier" is set to 1.0, the system extracts the two thresholds and performs comparability check, confirms that the level has both upward and downward channels, and there is no logical contradiction (such as no upward threshold for the top layer), finally combines the dynamic hotness evaluation value 5.3195 of data block '7A3F9B' with the upward migration threshold 5.0 and the downward migration threshold 1.0 of its current level to generate a comparison set of dynamic hotness evaluation value and migration threshold after comparability check.

[0053] According to the dynamic heat evaluation value after the comparability check and the migration threshold value set, the system starts a decision comparison process for each data block. Taking data block '7A3F9B' as an example, its set is {'F': 5.3195, 'up_threshold': 5.0, 'down_threshold': 1.0}. The system first performs the condition judgment of upward migration, that is, compares whether the dynamic heat evaluation value is greater than or equal to the upward migration threshold value. The difference value is 5.3195-5.0=0.3195. Since the difference value result is positive, it indicates that the heat of the data block has reached the upgrade standard. The system records the positive difference value 0.3195 and a 'positive' label of'suggested upward migration' together, and stops further comparison of the data block. For another data block 'F8G9H0', its dynamic heat evaluation value is 0.8, and the upward migration difference value is 0.8-5.0=-4.2, which is negative, indicating that it does not meet the upward migration condition. The system then performs the condition judgment of downward migration, calculates the difference value between it and the downward migration threshold value, which is 0.8-1.0=-0.2. The difference value result is negative, indicating that its heat is lower than the standard to remain in the current level. The system records the negative difference value -0.2 and a 'negative' label of'suggested downward migration'. For data blocks with dynamic heat evaluation values between the upward and downward migration threshold values, for example, the value is 4.0, the upward migration difference value is negative, and the downward migration difference value is positive, no label is generated. Through the bidirectional comparison and difference calculation of all data blocks, a difference label set of dynamic heat evaluation value and migration threshold value is generated.

[0054] Based on the difference label set of dynamic heat evaluation value and migration threshold value, the system traverses the set and makes a final migration decision according to the label state recorded therein. For any data block in the set that is assigned a 'positive' label, such as data block '7A3F9B', the system determines that it meets the condition of storage level upgrade because its dynamic heat evaluation value exceeds the upward migration threshold value of its current level. The system then generates an upgrade migration action record containing the unique identifier of the data block, the current storage level, and the 'MIGRATE_UP' instruction. For data blocks in the set that are assigned a 'negative' label, such as 'F8G9H0', the system determines that they meet the condition of storage level downgrade because their dynamic heat evaluation values are lower than the downward migration threshold value, thereby generating a downgrade migration action record containing the corresponding information of the 'MIGRATE_DOWN' instruction. For data blocks in the set that are not assigned any label, a 'NO_ACTION' record is generated. The system aggregates all the generated upgrade, downgrade or no-operation action records, and sorts them according to the migration priority (for example, the upgrade priority is higher than the downgrade), to form a structured instruction sequence, and generates a level migration decision signal.

[0055] The obtaining step of the data block migration instruction is: Based on the hierarchical migration decision signal, the migration action direction and the class identifier carried in the hierarchical migration decision signal are analyzed piece by piece, the target storage hierarchy information corresponding to the migration action direction identifier is extracted, and the data block target storage hierarchy identifier is generated; According to the data block target storage hierarchy identifier, the storage unit address and the port number corresponding to the target storage hierarchy identifier are matched, the capacity state and the current available space value of the target storage unit are obtained, it is judged whether the available space value of the target storage unit meets the size requirement of the current data block, and the data block target storage unit effectiveness confirmation result is formed; Based on the data block target storage unit effectiveness confirmation result, for the target storage unit whose available space value meets the data block size requirement, the data block migration instruction content is combined and constructed, including the unique identifier of the data block, the source storage unit position of the data block, the target storage unit address and the port number, and the data block migration priority information and the transmission mode mark are attached, and the data block migration instruction is generated.

[0056] Specifically, based on the hierarchical migration decision signal sequence generated in the previous step, the system starts an instruction analysis program, which processes the records in the sequence one by one. For a record, for example { 'block_id' : '7A3F9B', 'current_tier' : 'HDD_Tier', 'action' : 'MIGRATE_UP'}, the analysis program first extracts the migration action direction and the class identifier, that is, "MIGRATE_UP". Then, it queries a pre-constructed hierarchical mapping rule table that defines the storage hierarchy topology relationship. The table is manually created by the system administrator according to the physical deployment and performance gradient, which clearly defines the upper and lower levels of each level, for example, the table specifies that the upper level of "HDD_Tier" is "SSD_Tier" and the lower level is "Archive_Tier". According to the "MIGRATE_UP" identifier, the analysis program looks up the corresponding upper level in the rule table to determine the target storage hierarchy as "SSD_Tier". This information is output as a result, generating the data block target storage hierarchy identifier.

[0057] According to the data block target storage tier identifier, such as "SSD_Tier", the system then queries a detailed storage resource list, which is maintained in real time and contains the status of all physical storage units in the system, including their belonging tier, network address, access port, total capacity, used space, and current available space. The content is obtained by periodically sending query commands to each storage node through a background monitoring process and updating the summary. The system filters all storage units belonging to "SSD_Tier" in the list and determines an optimal target storage unit according to the preset load balancing strategy, such as selecting the unit with the highest current available space ratio. For example, the address of the selected SSD unit is 192.168.1.100, and the port number is 9000. The unit reports that its total capacity is 1TB, and the current available space is 200GB. Subsequently, the system queries the metadata information of the data block '7A3F9B' and obtains its size of 4MB. Then, the capacity is judged by comparing the available space 200GB with the data block size 4MB. 200GB is much larger than 4MB, so it is judged that the target storage unit has enough space to accommodate this data block. The system generates a confirmation result marked as "valid" based on this, forming the data block target storage unit validity confirmation result.

[0058] Based on the data block target storage unit validity confirmation result, i.e. the status marked as "valid", the system confirms that it can generate a formal migration instruction for the data block '7A3F9B'. If the confirmation result is "invalid", such as insufficient space in the target storage unit, the system will reselect another target unit or place the migration task in the waiting queue according to the preset retry strategy. After confirming the validity, the system starts to combine and construct the data block migration instruction. First, it integrates all necessary information: including the unique identifier of the data block '7A3F9B'; the data block source storage unit location obtained by querying the global storage mapping table, such as 'hdd_array_01 / volume_2 / offset_4096'; and the target storage unit address 192.168.1.100 and port number 9000 determined in the previous step. Then, the system will append two additional control markers. The first is the data block migration priority information, which is set according to the migration direction, such as "MIGRATE_UP" being assigned a high priority (value 1) and "MIGRATE_DOWN" being assigned a low priority (value 3). The second is the transmission mode marker. The system selects the optimal transmission protocol according to the network characteristics of the source and target storage tiers, such as marking the use of high-speed RDMA protocol for migration from HDD to SSD. Finally, all these information is encapsulated into a standardized data structure to generate the data block migration instruction.

[0059] The target migration path acquisition step is: Based on the data block migration instruction, a storage system management interface is called to obtain the source storage unit address where the data block currently locates and the target storage unit address after migration, and a source and target address set of data movement is formed. According to the source and target address set of data movement, through path query of the source storage unit address and the target storage unit address, each network node and link information associated in the path are extracted, a data transmission link node sequence from the source storage unit address to the target storage unit address is determined, and a target migration path is formed.

[0060] Specifically, based on the data block migration instruction generated in the previous step, for example, an instruction containing a data block '7A3F9B', a source location 'hdd_array_01 / volume_2 / offset_4096', and a target address '192.168.1.100:9000', the system sends the instruction to a unified storage system management interface. The interface is an API endpoint that provides standardized access services. It receives structured migration instructions as input and is responsible for parsing the field content therein. Specifically, the interface program directly extracts the value of the source storage unit location field from the data structure of the instruction, that is, 'hdd_array_01 / volume_2 / offset_4096', and takes it as the source address. At the same time, the value of the target storage unit address field '192.168.1.100' and the value of the port number field '9000' are extracted, and the two are combined into a network-reachable destination address '192.168.1.100:9000'. This process does not require secondary query or conversion of address information, but only structured data extraction. Finally, the parsed source address and target address are encapsulated into an address pair, for example, {'source': 'hdd_array_01 / volume_2 / offset_4096', 'destination': '192.168.1.100:9000'}. This operation is performed on each migration instruction in batch processing to form a source and target address set of data movement.

[0061] According to the address pair contained in the source and destination address set of data movement, such as {'source': 'hdd_array_01 / volume_2 / offset_4096', 'destination': '192.168.1.100:9000'}, the system calls the network path discovery service, which maintains a global network topology map, which is updated in real time through periodic running of link layer discovery protocol (LLDP) and border gateway protocol (BGP) route information collection, the interconnection relationship and link state of each switch and router in the network, the path query process adopts the improved Dijkstra algorithm, which is not only based on the number of hops, but also the real-time bandwidth utilization and delay of the link, the cost function is defined as cost=(0.6*normalized delay)+(0.4 / normalized available bandwidth), the weights 0.6 and 0.4 are pre-set according to the system's preference for low-latency data migration, the algorithm takes the access switch where the source address is located as the starting point and the access switch where the destination address is located as the end point, and calculates the path with the lowest cost in the network topology map, for example, the calculated path is: source server access switch SW-A1, core switch Core-SW1, aggregation switch AGG-SW2, destination server access switch SW-B3, and finally organizes each network device identifier on this path in order to form a data transmission link node sequence, that is, ['SW-A1', 'Core-SW1', 'AGG-SW2', 'SW-B3'], and the target migration path is obtained.

[0062] The acquisition step of the data retrieval optimization layout is: Based on the target migration path, the background daemon process is called to sequentially traverse the link node sequence recorded in the target migration path, and the data block pre-fetching on each node is performed, the data blocks are gradually cached to the intermediate nodes in the order of the data transmission link node sequence, the cache layout index table is updated to indicate the new layout position of the data blocks, and the data retrieval optimization layout is established.

[0063] Specifically, based on the target migration path, for example, ['SW-A1', 'Core-SW1', 'AGG-SW2', 'SW-B3'], a resident background daemon process in the system receives the path and the corresponding data block migration task, and the process then traverses each network node in the path in order. For intermediate nodes in the path (i.e., Core-SW1 and AGG-SW2), the daemon process queries the attributes of the nodes to determine whether they are designated nodes with caching capabilities. The designation is based on the hardware configuration of the nodes (e.g., having on-board solid state disks) and their centrality scores in the network topology. The centrality score is derived by calculating the betweenness centrality of the nodes. Nodes with centrality scores higher than a preset threshold of 0.75 are marked as cache points with high strategic value. The threshold is derived from historical network traffic analysis and is the minimum centrality value that can cover 80% of cross-regional data flows. If node Core-SW1 meets the conditions and its current cache available space is greater than the data block size, the daemon process will first prefetch the data block from the source to the cache of Core-SW1, and then forward it from Core-SW1 to the final destination. After the data block is landed on the final destination, the daemon process updates a global, distributed cache layout index table. The table is a key-value store, with the key being the data block identifier and the value being the list of all storage locations. For example, the entry of '7A3F9B' is updated to ['192.168.1.100:9000', 'Core-SW1_cache_address']. In this way, an optimized data retrieval layout is established.

[0064] The above is only a preferred embodiment of the present application, and does not limit the present application in other forms. Any skilled person in the art can modify or change the above disclosed technical content to equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments, without departing from the technical solution of the present application, still falls within the protection scope of the present application.

Claims

1. A method for fast data retrieval, characterized by, The method comprises the following steps: Based on the received data retrieval query request, locate the target data block and obtain the access type weight and access frequency count, generate the data block hotness value; Based on the data block hotness value, call the global clock to obtain the time difference, obtain the time-sensitive hotness weight, and perform operation based on the time-sensitive hotness weight and the data block hotness value to obtain the dynamic hotness evaluation value; Based on the dynamic hotness evaluation value, compare and judge with the migration threshold of the storage level where the data block is located to generate the level migration decision signal, match and determine the target storage level of the data block based on the level migration decision signal, and establish the data block migration instruction; Based on the data block migration instruction, construct the source and destination addresses of data movement, obtain the target migration path, and perform data block prefetching by the background daemon process based on the target migration path to establish data retrieval optimization layout.

2. The method of claim 1, wherein, The data block hotness value acquisition step is: Based on the received data retrieval query request, parse the data block identification field attached in the query request, match the data block identification field to the global storage mapping table, retrieve the access record entry set of the corresponding data block, form the access type field sequence and access timestamp field sequence; According to the access type field sequence and access timestamp field sequence, classify and count the access times in the set time window according to the access type, construct the time sequence set of each type of access, and extract the corresponding timestamp to calculate the change rate to obtain the access type count mapping table and the access type change rate index table; According to the access type count mapping table and the access type change rate index table, calculate the data block hotness value.

3. The method of claim 1, wherein, The time-sensitive hotness weight acquisition step is: Based on the data block hotness value, retrieve the hotness value generation timestamp contained in the generated record, and synchronously obtain the standard timestamp of the current time, calculate the difference in seconds between the current timestamp and the hotness value generation timestamp to form the hotness value time difference result; According to the hotness value time difference result, match the pre-defined time difference reference interval table, and sequentially compare each time difference value with the upper and lower limit ranges of the interval to determine the time decay section number where the hotness value time difference result is located, and extract the section identification information corresponding to the number to generate the time decay section identification; Based on the time decay section identification, read the numerical parameter content of the corresponding section from the set time-sensitive decay factor parameter table as the time-sensitive influence factor corresponding to the current data block to generate the time-sensitive hotness weight.

4. The method of claim 1, wherein, The dynamic hotness evaluation value acquisition step is: Based on the time-sensitive hotness weight, extract the hotness value corresponding to each data block to obtain the data block hotness value and the time-sensitive hotness weight; According to the data block hotness value and the time-sensitive hotness weight, construct a dynamic hotness input vector group, extract all data block indexes, label the data block position corresponding to each group of hotness parameters, and append the hotness saturation threshold and the time-sensitive influence factor as constant terms to form a hotness calculation input set; Based on the hotness calculation input set, calculate the dynamic hotness evaluation value.

5. The method of claim 1, wherein, The level migration decision signal acquisition step is: Based on the dynamic heat evaluation value, the migration threshold value of the storage level where the data block is located is extracted, the level type identifier corresponding to the migration threshold value is parsed piece by piece, and the dynamic heat evaluation value after comparability check and the migration threshold value are generated. The control set; According to the dynamic heat evaluation value after comparability check and the migration threshold value control set, the numerical difference value of the dynamic heat evaluation value of each data block and the corresponding storage level migration threshold value is calculated one by one, the difference value result between the dynamic heat evaluation value and the migration threshold value is recorded, and the positive and negative states of the difference value result are marked respectively, and the difference value mark set of the dynamic heat evaluation value and the migration threshold value is generated; Based on the difference value mark set of the dynamic heat evaluation value and the migration threshold value, the data block mark state with the numerical difference value greater than or equal to zero in the difference value mark set is extracted, and the difference value mark state is taken as the migration judgment basis. When the mark state is positive, the storage level up migration action is triggered, and when the mark state is negative, the storage level down migration action is triggered, and the direction and category of the migration action are recorded, and the level migration decision signal is generated.

6. The method for quick data retrieval according to claim 1, wherein, The acquisition step of the data block migration instruction is: Based on the level migration decision signal, the migration action direction and category identifier carried in the level migration decision signal are parsed piece by piece, the target storage level information corresponding to the migration action direction identifier is extracted, and the data block target storage level identifier is generated; According to the data block target storage level identifier, the storage unit address and port number corresponding to the target storage level identifier are matched, the capacity state and current available space value of the target storage unit are obtained, it is judged whether the available space value of the target storage unit meets the size requirement of the current data block, and the data block target storage unit effectiveness confirmation result is formed; Based on the data block target storage unit effectiveness confirmation result, for the target storage unit whose available space value meets the size requirement of the data block, the data block migration instruction content is combined and constructed, including the unique identifier of the data block, the source storage unit position of the data block, the target storage unit address and port number, and the data block migration priority information and transmission mode mark are attached, and the data block migration instruction is generated.

7. The method for quick data retrieval according to claim 1, wherein, The acquisition step of the target migration path is: Based on the data block migration instruction, the storage system management interface is called, the source storage unit address where the data block is currently located and the destination storage unit address after migration are obtained, and the source and destination address set of data movement is formed; According to the source and destination address set of data movement, the path query of source storage unit address and destination storage unit address is carried out, the network nodes and link information associated in the path are extracted, the data transmission link node sequence from the source storage unit address to the destination storage unit address is determined, and the target migration path is formed.

8. The method for quick data retrieval of claim 1, wherein, The acquisition step of the data retrieval optimization layout is: Based on the target migration path, the background daemon process is called, the link node sequence recorded in the target migration path is traversed in turn, the data block prefetching on each node is carried out, the data block is gradually cached to the intermediate node according to the order of the data transmission link node sequence, the cache layout index table is updated to indicate the new layout position of the data block, and the data retrieval optimization layout is established.

9. The data quick search system of the data quick search method according to any one of claims 1 to 8, characterized by, It includes: The query identification module retrieves a query request based on the received data, locates to a target data block, and acquires an access type weight and an access frequency count, and generates a data block heat value. The heat evaluation module acquires a time difference by calling a global clock based on the data block heat value, acquires a time effectiveness heat weight, and performs an operation based on the time effectiveness heat weight and the data block heat value to obtain a dynamic heat evaluation value. The hierarchical decision module compares and judges the dynamic heat evaluation value with a migration threshold of a storage hierarchy where the data block is located, generates a hierarchical migration decision signal, matches and determines a target storage hierarchy of the data block based on the hierarchical migration decision signal, and establishes a data block migration instruction. The data prefetching module constructs a source and a destination address of data movement based on the data block migration instruction, acquires a target migration path, performs data block prefetching by a background daemon process based on the target migration path, and establishes a data retrieval optimization layout.

Images