Massive small file reading method of hadoop based on file merging and dual elimination mechanism of heat and time limit

Abstract

The application belongs to the technical field of computers and discloses a mass small file reading method of Hadoop based on file merging and a heat and time dual elimination mechanism, which is as follows: a small file reading request input by a user is received, and it is determined whether the file of the reading request is a small file; if not, a request is sent to a NameNode in a HDFS system, the NameNode reads corresponding files from DataNodes according to metadata information; if yes, a Redis cache module is inquired according to a complete path of the small file, if the small file requested to be read is cached in the Redis cache module, the small file is directly returned to the user; the Redis cache module caches part of hot small files according to a cache update strategy, the hot small files are obtained by a small file access prediction module through a heat and time dual elimination mechanism based on a heat calculation formula; if the small file requested to be read is not cached in the Redis cache module, a request is sent to the NameNode in the HDFS system, and the NameNode reads corresponding files from the DataNodes according to the metadata information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a mass small file reading method of Hadoop based on file merging and heat and time dual elimination mechanism. BACKGROUND

[0002] In the big data processing ecosystem, Hadoop has become a key technology platform for storing and processing massive data due to its distributed architecture and strong scalability. However, as the data scale continues to expand and the data types become increasingly complex, the processing of mass small files has gradually become a bottleneck for the performance improvement of Hadoop system. Taking the e-commerce industry as an example, the number of small files such as daily generated commodity comments and order details can reach millions or even tens of millions. When these small files are stored in the Hadoop Distributed File System (HDFS), each small file needs to occupy a certain amount of memory space in the NameNode to store its metadata (such as file name, file size, storage location, etc.). When the number of small files is large, the memory pressure of the NameNode increases dramatically, resulting in slower system response speed and even freezing phenomenon.

[0003] At the same time, the traditional Hadoop method in reading mass small files, frequent disk I / O operation is one of the main reasons for low performance. Small file data is small, each read needs to be disk seek, head positioning and other operations, the overhead of these operations is constantly accumulated when a large number of small files are read, so that disk I / O becomes the performance bottleneck of the whole system. For example, the time spent on reading 100 10KB small files is much longer than reading one 1MB large file, even if the total amount of data is the same. This is because a large file can be read at a time, while a small file needs to be read multiple times, increasing the system waiting time.

[0004] In addition, the NameNode is the core component of HDFS, responsible for managing the namespace and file metadata of the file system. Mass small files mean mass metadata storage requirements, which makes the memory consumption of the NameNode rise sharply. At the same time, when processing file reading requests, the NameNode needs to frequently query and process these metadata, resulting in excessive load and slow processing speed. For example, when the NameNode memory stores the metadata of millions of small files, it may take several seconds or even longer to find the metadata of a specific small file, seriously affecting the efficiency of file reading. SUMMARY

[0005] In order to overcome the above problems of low efficiency in reading mass small files by traditional Hadoop method, the present application provides a mass small file reading method of Hadoop based on file merging and heat and time dual elimination mechanism.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is:

[0007] A mass small file reading method of Hadoop based on file merging and heat and time dual elimination mechanism, the method is suitable for HDFS system with data merging module and Redis cache module, the method comprises the following steps:

[0008] Receiving a small file reading request input by a user, determining whether the file of the reading request is a small file;

[0009] If not, sending a request to the NameNode in the HDFS system, the NameNode reads the corresponding file from the DataNode according to the metadata information;

[0010] If yes, querying the Redis cache module according to the complete path of the small file, if the small file to be read is cached in the Redis cache module, the small file is directly returned to the user; the Redis cache module caches part of the heat small files according to the cache update strategy, the heat small files are predicted by a small file access prediction module constructed by using a heat calculation formula based on the dual elimination mechanism of hot spot and time limit;

[0011] If the small file to be read is not cached in the Redis cache module, sending a request to the NameNode in the HDFS system, the NameNode reads the corresponding file from the DataNode according to the metadata information.

[0012] Preferably, the heat calculation formula based on the dual elimination mechanism of hot spot and time limit comprises the following:

[0013]

[0014] Wherein, a represents the attenuation coefficient, 0<α<1; countPeriod represents the heat calculation period; visitCount represents the number of times the small file is accessed in the current heat calculation period; heat n-1 represents the historical heat of the small file.

[0015] Preferably, before receiving the small file reading request input by the user, the method further comprises: initializing the HDFS system;

[0016] The initialization comprises:

[0017] Configuring network parameters, cache parameters, the cache parameters including cache capacity, cache expiration time;

[0018] The cache update strategy engine is started, a pre-trained small file access prediction model is loaded, and relevant parameters and variables are initialized, including a historical data storage location and a small file access prediction model update frequency.

[0019] The Redis cache module is initialized, an initial size of the cache space is set, and relevant parameters of the cache update strategy are configured; the relevant parameters of the cache update strategy include a preloading time window and a preloading file type.

[0020] Preferably, the cache update strategy comprises:

[0021] When the read request is executed, only the number of times of accessing files in the hotness calculation period is recorded, until the number of requests reaches countPeriod, i.e., the hotness calculation period ends, and the service process triggers the update and replacement operation of the cache;

[0022] At the same time, the hotness values of all files are calculated according to the hotness calculation formula, and the files and metadata with the top-K hotness values are cached into the Redis cache module by using the "access and insert" strategy;

[0023] When the cache space of the Redis cache module is full, the file with the minimum hotness value is selected according to the hotness values of the files, and the file with the top-K hotness values is kept in the cache, so as to realize dynamic update and optimization of the cache data.

[0024] Further, the cache update strategy further comprises: preloading the files with the top-K hotness values into the Redis cache module during a system idle period or a low-load period.

[0025] Preferably, the data merging method adopted by the data merging module is as follows:

[0026] First, the historical access logs of the small files are analyzed to obtain the access conditions of each small file by the user;

[0027] The overall correlation of a certain small file with all files in a merging queue is calculated.

[0028] When the overall correlation reaches a preset merging threshold, the small file is merged into the merging queue and stored in the HDFS by using the API provided by the Hadoop to call the MapFile; the file structure of the MapFile is composed of an index and data, which are used to store index files and data, respectively.

[0029] Further, the calculation of the overall correlation of a certain small file with all files in a merging queue comprises:

[0030] Correlation(f, fi) = {(f, fi) | P(fi|f) e (th1, th2) * P(ffi) e (th3, th4)} (1)

[0031]

[0032] In formula 1, Correlation(f, fi) represents the correlation between file f and file fi; (f, fi) represents a relationship binary tuple composed of f and fi; P(fi|f) represents the probability of file fi being accessed within a period of time after file f is accessed; P(ffi) represents the probability of a user accessing both f and fi within a period of time; and th represents different thresholds, which are all within the range of (0, 1).

[0033] In formula 2, Correlation(f, q) represents the overall correlation between the current small file f and all files in a certain merging queue q; and n represents the number of merging queues.

[0034] Preferably, the method further comprises a file writing process, which comprises:

[0035] receiving a file writing request sent by a user, and determining whether the file to be written is a small file;

[0036] If the file is a small file, the data merging module is entered for data merging, and the merged small file is transmitted to the HDFS for storage;

[0037] If the file is not a small file, the HDFS system is directly entered for storage operation.

[0038] Preferably, the method further comprises a file deletion process, which comprises:

[0039] receiving a file deletion request sent by a user, and determining whether the file to be deleted is a small file;

[0040] If the file is not a small file, the HDFS system is directly entered for deletion operation;

[0041] If the file is a small file, the Redis cache module is queried according to the complete path of the file; if the Redis cache module has the file to be deleted, the small file cached in the Redis cache module is deleted;

[0042] If the Redis cache module does not have the file to be deleted, a deletion request is sent to the NameNode in the Hadoop cluster; the NameNode and the DataNode are matched and interacted to find the corresponding small file, and a deletion request is sent for deletion operation; and the information of the deleted small file is returned to the user.

[0043] Preferably, the method further comprises:

[0044] When a small file in Hadoop is updated, deleted or newly added, the corresponding information is obtained through a message queue or an event listening mechanism;

[0045] The Redis cache module synchronously updates, deletes or newly adds the small file cached in the Redis cache module according to the received information.

[0046] Compared with the prior art, the present application has the beneficial effects that:

[0047] The present application optimizes the distributed system HDFS, combines a small file cache update strategy, and introduces a small file access prediction module in the Redis cache module, aiming to significantly improve the reading speed of massive small files, greatly reduce the reading delay, comprehensively enhance the performance and adaptability of the Hadoop system in the massive small file processing scenario, and meet the urgent needs of various industries for real-time processing of big data.

[0048] The present application deeply penetrates the core field of big data storage and reading technology, and focuses on making innovative breakthroughs in the performance dilemma faced by the Hadoop platform when processing massive small file reading. In the current era of accelerating digital transformation, the amount of data in various industries is growing explosively, and massive small file data widely exists in many business scenarios such as e-commerce transaction records, social platform user dynamics, and financial institution transaction details. Hadoop, as the mainstream platform for big data storage and processing, its reading performance of massive small files directly affects the efficiency of data value mining and the smoothness of business operation. The present application is expected to provide efficient and reliable technical support for the above-mentioned industries and related fields, and promote the in-depth development of big data applications. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a step flow chart of the massive small file reading method of Hadoop based on file merging and the dual elimination mechanism of heat and time efficiency of the present application.

[0050] Figure 2 is a flow chart of the cache management process of the present application.

[0051] Figure 3 is a flow chart of the file merging process of the present application.

[0052] Figure 4 is a flow chart of the file writing process of the present application.

[0053] Figure 5 is a flow chart of the file deletion process of the present application. DETAILED DESCRIPTION

[0054] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application. The present application will be described in detail below with reference to the drawings and specific embodiments.

[0055] It should be understood that the terms "comprising" and "including" as used in the specification, indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0056] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0057] It should be further understood that the term "and / or" used in the present application specification means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0058] Embodiment 1

[0059] As shown in Figure 1 A mass small file reading method of Hadoop based on file merging and hotness and timeliness double elimination mechanism, the method is suitable for HDFS system with data merging module and Redis cache module, the method comprises the following steps:

[0060] Receiving a small file reading request input by a user, determining whether the file of the reading request is a small file;

[0061] If not, sending a request to the NameNode in the HDFS system, the NameNode reads the corresponding file from the DataNode according to the metadata information;

[0062] If yes, querying the Redis cache module according to the complete path of the small file, if the small file to be read is cached in the Redis cache module, directly returning the small file to the user; the Redis cache module caches part of hotness small files according to the cache update strategy, the hotness small files are predicted by a small file access prediction module constructed by using a hotness calculation formula based on hot spot and timeliness double elimination mechanism;

[0063] If the small file requested to be read is not in the Redis cache module, a request is sent to the NameNode in the HDFS system, and the NameNode reads the corresponding file from the DataNode according to the metadata information.

[0064] In a specific embodiment, in order to improve the access efficiency of small files in the HDFS system, an intelligent cache mechanism is designed and implemented. By combining the small file access prediction module with the cache update strategy, part of the hot small files is cached in the Redis cache module, which can reduce the time-consuming of operations such as searching metadata in the NameNode and reading Block in the DataNode, thereby effectively improving the access efficiency of small files in the HDFS.

[0065] Based on the reasonable assumption that "the data recently accessed is most likely to be repeatedly accessed in the near future", the application designs a unique double eviction mechanism. By periodically accumulating the number of times the cached files are accessed and converting it into a heat value, accurate screening of hot data is realized.

[0066] The heat calculation formula of the double eviction mechanism based on hot spots and time efficiency includes the following:

[0067]

[0068] Wherein, alpha represents the decay coefficient, 0 < alpha < 1; countPeriod represents the heat calculation period; visitCount represents the number of times the small file is accessed in the current heat calculation period; heat n-1 represents the historical heat of the small file.

[0069] The heat calculation formula skillfully balances the weight of the current access count and the historical heat in calculating the file heat. The larger alpha is, the greater the weight of the recent access in the data access heat, and the smaller the influence of the historical access record on the data heat; on the contrary, the smaller alpha is, the greater the influence of the historical heat. In the actual running process, the historical heat of the file will decay at a rate of (1-alpha) coefficient in each calculation period; after multiple iterations, the influence of the early cumulative heat on the data heat gradually decreases, thereby ensuring that the heat calculation can timely reflect the real-time access heat of the file.

[0070] In a specific embodiment, in addition to the small file access prediction module predicted by the heat calculation formula of the double eviction mechanism based on hot spots and time efficiency, the intelligent cache mechanism also includes a cache update strategy.

[0071] The cache update strategy, as shown in Figure 2 , includes:

[0072] In order to reduce the system overhead caused by the heat calculation, when the cached service process executes the read request, only the number of times of accessing the file in the heat calculation period is recorded, and the cache data is not replaced immediately, until the number of requests reaches countPeriod, that is, the end of the heat calculation period, the service process triggers the update and replacement operation of the cache;

[0073] Meanwhile, the heat values of all files are calculated according to the heat calculation formula, and the files and metadata with the heat values ranked TOP-K are cached into the Redis cache module by using the "access and insert" strategy; when the heat values ranked TOP-K are determined, the size limitation of the cache space is fully considered, and the heat threshold is calculated to ensure the reasonable use of the cache space. In the initial stage of the system, since the cache space is largely idle, the "access and insert" strategy is used to insert all accessed files into the cache, so as to quickly establish the initial cache dataset.

[0074] When the cache space of the Redis cache module is full, the file with the minimum heat value is selected to be evicted from the cache according to the heat value of the file, and the file with the heat value ranked TOP-K is retained in the cache, so as to realize the dynamic update and optimization of the cache data. The cache replacement strategy based on the heat value of the application starts to play a role, according to the heat value of the file, the small file ranked after TOP-K is selected as the "sacrifice" to be evicted from the cache, and the small file with a high heat value is retained in the cache, so as to realize the dynamic update and optimization of the cache data.

[0075] In a specific embodiment, the cache update strategy further comprises: using the system idle period or low load period to pre-load the file with the heat value ranked TOP-K into the Redis cache module.

[0076] The application fully utilizes the resources of the system idle period or low load period by introducing the cache pre-loading strategy, and determines the pre-loaded hot files by analyzing the system access mode through the small file access prediction module. For example, in a news information platform, the morning is the peak period of user access to news small files, and the system can load the news small files and metadata that may be accessed a lot in the day in advance into the cache according to the historical access data and heat value prediction in the low load period in the early morning. In this way, when the morning high concurrency access occurs, the data requested by the user can be directly obtained from the cache, so that the response speed of the system and the user experience are greatly improved.

[0077] The present application collects and stores historical access data of small files, including access time, access frequency, and other rich information. Advanced machine learning algorithms such as decision trees and neural networks are used to deeply mine and analyze these historical data. Combined with the heat calculation formula based on the dual elimination mechanism of heat and time, a highly accurate small file access prediction model is constructed. Taking a social media platform as an example, by analyzing the user's access frequency and pattern of picture small files and text small files in different time periods (such as weekdays, weekends, holidays) and different activities (such as posting dynamics, browsing friend dynamics, liking and commenting), a model that can accurately predict the probability of small file access in different scenarios is trained. This model not only considers the impact of time factors on file access, but also combines user behavior characteristics, improving the accuracy and reliability of the prediction.

[0078] After building the small file access prediction module, as time goes by and new access data accumulates, data access patterns may change. In order to keep the small file access prediction module always accurate, it needs to be retrained and optimized regularly. When the platform introduces new functions (such as short video function) or the business scenario changes significantly (such as user group structure adjustment), the input features and training parameters of the small file access prediction module need to be adjusted in time. For example, after introducing the short video function, the short video related access data is included in the training range of the small file access prediction module, and the feature weight is adjusted to adapt to the new access mode, ensuring that the small file access prediction module can accurately predict the access trend of small files, providing a reliable basis for intelligent prefetching.

[0079] In a specific embodiment, before receiving the user input small file reading request, the method further comprises initializing the HDFS system;

[0080] The initialization includes:

[0081] Configure network parameters, cache parameters, including cache capacity, cache expiration time.

[0082] The application carefully deploys and configures the Redis cache module in the Hadoop cluster, ensures the establishment of stable and efficient connection channels between the Redis cache module and each node of the Hadoop cluster. By reasonably configuring network parameters and cache parameters (such as cache capacity, cache expiration time, etc.), the data can be quickly transmitted and interacted between the Hadoop cluster and the Redis cache, providing a solid foundation for subsequent cache operations. The Redis cache module is also used to receive and process write requests initiated by the client Client, and according to the cache storage strategy, the cache is stored in the Redis cache module for management, which will greatly improve the read request initiated by the client Client. At the same time, the Redis cache module is also responsible for managing all meta information data in the small file. The Redis cache module works with the Hadoop cluster, and the Redis cache module shares the concurrent pressure brought by the massive files, and greatly improves the reading efficiency of the hot files.

[0083] The cache update strategy engine is started, and a pre-trained small file access prediction model is loaded; relevant parameters and variables are initialized, including the storage location of historical data (specifying the database table or file path for storing historical access data), the update frequency of the small file access prediction model (set to once a week or dynamically adjusted according to business needs). Ensure that the prefetch decision engine can run accurately and provide strong support for intelligent prefetching.

[0084] The Redis cache module is initialized, the initial size of the cache space is set, and the related parameters of the cache update strategy are configured; the related parameters of the cache update strategy include the preloading time window (for example, set 2am-5am as the preloading time window), the preloading file type (according to business needs, such as specifying product pictures, user evaluation files, etc. Type). Through the initialization of the Redis cache module, the effective management of the cache and the preloading of the hot files are well prepared.

[0085] In the initial stage of the system, due to the sufficient free space of the cache, the strategy of "accessing and inserting" is adopted, and all accessed files and their metadata are inserted into the Redis cache module for caching, so as to quickly build an initial cache dataset. When the Redis cache module is gradually filled, a double elimination mechanism based on heat and time and a cache preloading strategy are combined to dynamically replace the hot files and their metadata in the cache. After each file access operation, the cache service process records the access frequency of the file, and when the heat calculation period is reached, the file heat is recalculated, and the data in the cache is replaced according to the heat ranking, to ensure that the cache always stores the hot files that are most likely to be accessed again. The hot files are stored in the Redis cache module. The Redis cache module is a Key-Value type database, and the Key value is the complete path of the stored file, and the Value value is the stored file content. When the data is retrieved, the Redis cache module can be accessed first to determine whether there is data caching, and if there is, the corresponding data is returned directly, and if there is not, the data in the HDFS is accessed.

[0086] In a specific embodiment, the data merging module adopts the following data merging method:

[0087] First, analyze the historical access log of the small files to obtain the access situation of each small file by the user;

[0088] Calculate the overall relevance of a small file to all files in a merge queue;

[0089] When the overall relevance reaches a preset merging threshold, the small file is merged into the merge queue and stored in the HDFS by the API provided by Hadoop to call the MapFile. The file structure of the MapFile consists of index and data, which are used to store index files and data, respectively.

[0090] The calculation of the overall relevance of a small file to all files in a merge queue includes:

[0091] Correlation(f, fi) = {(g, fi) | P(fi | g) e (th1, th2) * P(ffi) e (th3, th4)} (1)

[0092]

[0093] Correlation(f, fi) represents the correlation between file f and file fi; (f, fi) represents the relationship between the two files; P(fi|f) represents the probability of file fi being accessed within a period of time after file f has been accessed; P(ffi) represents the probability of both file f and file fi being accessed within a period of time; and th represents different thresholds, all of which are within the range of (0, 1).

[0094] In formula 2, Correlation(f, q) represents the overall correlation between the current small file f and all files in a certain merge queue q; and n represents the number of merge queues.

[0095] The data structure required by the merging method is described as follows:

[0096] Small file f: represents the current small file to be stored.

[0097] Merge queue q: responsible for temporarily storing small files, and merging into a MapFile when the size reaches the threshold of merging.

[0098] Merge queue set qList: the set of merge queues q, containing n q.

[0099] inuseList: a subset of qList, representing the set of non-empty and non-merged q.

[0100] fitList: a subset of inuseList, representing the set of q in inuseList with remaining space greater than the current small file f.

[0101] candidateList: a subset of fitList, representing the set of q in fitList with correlation exceeding the threshold for the current small file f.

[0102] The specific merging algorithm steps are as follows, as shown in Figure 3 :

[0103] a. Initialize the merge queue set qList with a capacity of n.

[0104] b. For the current small file f, first check inuseList. If inuseList is not empty, select all q that meet the conditions (i.e., q that can accommodate the current f) and add them to fitList; if inuseList is empty, f enters any empty q, and f enters inuseList.

[0105] c. If fitList is not empty, traverse the queue in fitList. For each queue qi, calculate the average correlation of f in q, Correlation(f, q). If Correlation(f, q) > fitThreshold, f enters q; if Correlation(f, q) < fitThreshold, q is removed from fitList and enters candidateList.

[0106] d. candidateList is set to balance the correlation between files and the distribution of files. If only the correlation is considered, when the correlation of the whole files is not close, the space will be wasted. At this time, a compromise is needed: select the queue q with the highest average correlation for file f in candidateList m , judge Correlation(f, q m ), if the value exceeds SecondThreshold, f enters q m ; otherwise, do not consider the correlation, and f enters a new queue.

[0107] e. If fitThreshold is empty, it means that there is no merge queue that can accommodate the current small file at this time. Check the queue q with the smallest remaining space in inuseList m , if the size of q m reaches the merge threshold (128MB of HDFS block size), the files in q are merged into MapFile.

[0108] For the merge queue that meets the merge condition, the small files in it will be merged and stored in the form of MapFile provided by Hadoop. The file structure of MapFile consists of index and data, which are used to store index files and data respectively. The key value of each record is the file name, which is stored as an index by index; the value stores the file content. Compared with the other two merge storage methods provided by Hadoop, MapFile has higher access efficiency.

[0109] In a specific embodiment, the method further comprises a file writing process, the file writing process comprising:

[0110] receiving a file writing request sent by a user, and determining whether the file to be written is a small file;

[0111] if it is a small file, entering a data merging module to perform data merging, and transmitting the merged small file to HDFS for storage;

[0112] If not, directly enter the HDFS system for storage operation. Specifically, send a write request to the NameNode in the HDFS system, the NameNode receives the file write signal and finds the corresponding data node DataNode, and stores the file in the DataNode. The DataNode receives and confirms the data information initiated by the NameNode to write the file, and finally returns the write report to the NameNode. The write result is returned to the user, and then the data stream is closed, and the write operation is completed. The specific process is shown in Figure 4 .

[0113] In a specific embodiment, the method further comprises a file deletion process, which comprises:

[0114] receiving a user-sent file deletion request; determining whether the deleted file is a small file;

[0115] If not, directly enter the HDFS system for deletion operation;

[0116] If yes, query the Redis cache module according to the file complete path; if the Redis cache module has the file to be deleted, delete the small file cached in the Redis cache module;

[0117] If the Redis cache module does not have the file to be deleted, send a deletion request to the NameNode in the Hadoop cluster; the NameNode and the DataNode are matched and interacted, the corresponding small file is found, the deletion request is sent for deletion operation; and the small file deletion information is returned to the user. The specific process is shown in Figure 5 .

[0118] Preferably, the method further comprises:

[0119] When the small file in Hadoop is updated, deleted or newly added, the corresponding information is obtained through the message queue or event listening mechanism to timely capture these operation events;

[0120] The Redis cache module synchronously updates, deletes or newly adds the small file cached in the Redis cache module according to the received information; thereby ensuring the consistency of the cache data and the Hadoop storage system data, and avoiding reading errors caused by inconsistent data.

[0121] In this embodiment, the data in the cache can be periodically (e.g., weekly) subjected to a comprehensive integrity check and repair. By comparing with the original data in the Hadoop storage system, it is detected whether there is loss, damage or inconsistency in the cache data. If problems are found, appropriate repair measures are taken in time, such as reloading data from the Hadoop storage system, updating error data in the cache, etc., to ensure the accuracy and availability of the cache data.

[0122] This embodiment can also analyze and optimize the cache management strategy according to the system's operation log and performance monitoring data. For example, by analyzing the cache hit rate, disk I / O load and other indicators in different time periods, the pre-fetch time window is optimized, and parameters such as the heat calculation period and decay coefficient are adjusted to improve the utilization of cache space and the overall performance of the system, so that the system can always run efficiently and stably in the scenario of massive small file reading.

[0123] This embodiment greatly improves the efficiency of massive small file reading based on Hadoop through the synergistic effect of multi-element cache optimization, double elimination mechanism and cache preloading strategy. In actual tests, compared with the traditional Hadoop reading method, the reading delay is reduced by more than 80%, and the system response speed is improved by 3 to 10 times. In the real-time analysis of financial transaction data, a large number of transaction detail small files can be quickly read to provide timely and accurate data support for risk assessment, market trend analysis, etc., meeting the strict requirements of real-time business.

[0124] The present application improves the utilization of cache space by combining cache update strategy with small file access prediction module, reduces the redundancy and invalid occupation of cache data. By dynamically eliminating files with low heat value according to file heat and time, it ensures that the most valuable hotspot data is always stored in the cache, reduces the storage cost and resource consumption of the system. At the same time, it reduces the disk I / O operation in HDFS, reduces the disk load, prolongs the service life of the disk, further improves the overall performance and stability of the system, and enhances the competitiveness of the Hadoop system in the scenario of massive small file processing.

[0125] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not a limitation on the embodiments of the present application. Any modification, equivalent replacement and improvement within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A Hadoop-based mass small file reading method based on file merging and a hotness and timeliness dual elimination mechanism, the method being applicable to a HDFS system having a data merging module and a Redis cache module, and characterized in that it comprises the following steps: 1) a file merging module is used to merge small files into a large file; 2) a Redis cache module is used to store the merged large file; 3) a hotness and timeliness dual elimination mechanism is used to eliminate the large file; and 4) a file reading module is used to read the large file. The method comprises the following steps: Receiving a user input small file reading request, determining whether the file of the reading request is a small file; If not, sending a request to the NameNode in the HDFS system, and the NameNode reads the corresponding file from the DataNode according to the metadata information; If yes, querying the Redis cache module according to the complete path of the small file, if the small file to be read is cached in the Redis cache module, the small file is directly returned to the user; the Redis cache module caches some hot small files according to the cache update strategy, and the hot small files are predicted by a small file access prediction module constructed by using a hot spot and time-based double elimination mechanism hotness calculation formula; If the small file to be read is not cached in the Redis cache module, a request is sent to the NameNode in the HDFS system, and the NameNode reads the corresponding file from the DataNode according to the metadata information; The cache update strategy comprises: During the execution of the reading request, only the number of times of accessing the file in the hotness calculation period is recorded, until the request number reaches countPeriod, that is, the hotness calculation period ends, and the service process triggers the update replacement operation of the cache; At the same time, the hotness values of all files are calculated according to the hotness calculation formula, and the files with the top-K hotness values and their metadata are cached into the Redis cache module according to the "access and insert" strategy; When the cache space of the Redis cache module is full, the file with the smallest hotness value is selected according to the hotness value of the file, and the file with the top-K hotness value is kept in the cache, so as to realize the dynamic update and optimization of the cache data; The cache update strategy further comprises: using the system idle period or low load period to pre-load the files with the top-K hotness values into the Redis cache module.

2. The method of claim 1, wherein the method is characterized in that: The hotness calculation formula based on the double elimination mechanism of hot spot and time efficiency comprises the following: wherein a represents an attenuation coefficient, 0 < a < 1; countPeriod represents a heat calculation period; visitCount represents the number of times the small file is accessed in the current heat calculation period; heat n-1 represents the historical heat of the small file. 3.The method of claim 1, wherein the method further comprises: determining whether the file is a small file; and if the file is a small file, reading the file from the Hadoop system based on a file merge and a hotness and time expiration dual elimination mechanism. Before receiving the user input small file reading request, the method further comprises: initializing the HDFS system; The initialization comprises: Configuring network parameters and cache parameters, the cache parameters comprising cache capacity and cache expiration time; Starting the cache update strategy engine, loading the pre-trained small file access prediction model; initializing related parameters and variables, including historical data storage location, small file access prediction model update frequency; Initializing the Redis cache module, setting the initial size of the cache space, and configuring related parameters of the cache update strategy; the related parameters of the cache update strategy comprise a pre-loading time window and a pre-loading file type.

4. The method of claim 1, wherein the method further comprises: determining whether the file is a small file; and if the file is a small file, then: determining whether the file is a hot file; and if the file is a hot file, then: determining whether the file is a cold file; and if the file is a cold file, then: deleting the file. The data merging method adopted by the data merging module is as follows: First, analyze the historical access log of the small file to obtain the access situation of each small file by the user; Calculate the overall correlation between a certain small file and all files in a certain merging queue; When the overall correlation reaches a preset merging threshold, the small files are merged into a merging queue and stored in HDFS by a way of calling an API of MapFile provided by Hadoop; The file structure of the MapFile is composed of two parts of index and data, which are used to store index files and data respectively.

5. The Hadoop-based mass small file reading method based on file merging and a hotness and time dual elimination mechanism according to claim 4, characterized in that: The calculation of the overall correlation of a small file with all files in a merging queue includes: Correlation(f, fi)={(f, fi)|P(fi|f)∈(th1,th2)*P(ffi)∈(th3,th4)} Wherein, Correlation(f, fi) in formula 1 represents the correlation degree between file f and file fi; (f, fi) represents a relationship binary tuple composed of f and fi; P(fi|f) represents the probability of accessing file fi within a period of time after file f is accessed; P(ffi) represents the probability of accessing both f and fi within a period of time; th1, th2, th3 and th4 represent different threshold values, and the value range is within the interval (0, 1); Correlation(f, q) in formula 2 represents the overall correlation degree of the current small file f with all files in a merging queue q; n represents the number of the merging queue.

6. The Hadoop-based mass small file reading method based on file merging and a hotness and time dual elimination mechanism according to claim 4, characterized in that: The method further includes a file writing process, which includes: Receiving a file writing request sent by a user, and determining whether the written file is a small file; If it is a small file, entering a data merging module to perform data merging, and transmitting the merged small file to HDFS for storage; If it is not a small file, directly entering the HDFS system to perform storage operation.

7. The Hadoop-based mass small file reading method based on file merging and a hotness and time dual elimination mechanism according to claim 1, characterized in that: The method further includes a file deletion process, which includes: Receiving a file deletion request sent by a user, and determining whether the deleted file is a small file; If not, directly entering the HDFS system to perform deletion operation; If yes, querying the Redis cache module according to the complete path of the file; if the Redis cache module has the file to be deleted, performing deletion operation on the small file cached in the Redis cache module; If the Redis cache module does not have the file to be deleted, sending a deletion request to the NameNode in the Hadoop cluster; the NameNode and the DataNode are matched and interacted to find the corresponding small file, and a deletion request is sent to perform deletion operation; and the information of the deleted small file is returned to the user.

8. The Hadoop-based mass small file reading method based on file merging and a hotness and time dual elimination mechanism according to claim 1, characterized in that: The method further includes: When the small file in Hadoop is updated, deleted or newly added, the corresponding information is obtained through a message queue or an event listening mechanism; The Redis cache module synchronously updates, deletes or newly adds the small file cached in the Redis cache module according to the received information.

Images