Mass small file reading optimization method, system, equipment and medium

Abstract

The invention relates to the technical field of big data, and discloses a massive small file reading optimization method, system and device and a medium, and the method comprises the following steps: scanning small files in a distributed file system by using a Spark calculation engine to obtain metadata information of the small files; based on the metadata information, small files are classified, and the small files with the same type and data relevance are classified into one group; the resource states of cluster nodes are monitored in real time, the resource states comprise the processor utilization rate, the memory occupancy rate and the I / O load, the load score of each node is calculated, and small file task quotas are dynamically allocated; executing small file merging operation in parallel by utilizing a Spark memory calculation engine to generate a merged file containing a plurality of original small files; creating and maintaining index information in an external database, and recording the position and attribute of each original small file in the combined file; and providing a read-write interface through the special service layer, and positioning and accessing specific small file contents in the combined file.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular, to a method, system, device, and medium for optimizing the reading of a large number of small files. Background Art

[0002] In today's big data era, the data volume in all walks of life shows an explosive growth, and the problem of a large number of small files has become increasingly prominent. Especially in the Hadoop Distributed File System (HDFS), the cornerstone of big data processing, the low efficiency of small file processing has become a key bottleneck restricting the performance of big data applications. Taking industries such as e-commerce, finance, and healthcare as examples, a large number of daily transaction records, user behavior data, medical images, etc. are often stored in the form of small files, with the number often reaching millions or even tens of millions.

[0003] As the core component of big data storage, the architecture design of HDFS exposes inherent limitations when dealing with a large number of small files. The NameNode node of HDFS is responsible for managing the metadata of the entire file system, including file names, directory structures, file block location information, etc. For each file, even a small file, corresponding metadata information needs to be maintained in the NameNode. When the number of small files surges, the memory consumption of the NameNode rises sharply, resulting in a memory bottleneck, seriously affecting the performance and stability of the NameNode, and even possibly causing the system to crash. For example, millions of small files may occupy gigabytes of memory on the NameNode node, which is a huge burden for a cluster environment with limited resources.

[0004] On the other hand, HDFS and the traditional MapReduce computing framework are more focused on processing large files in design, and the efficiency of frequent I / O operations for small files is low. Reading a small file usually requires multiple disk addressings and network transmissions. Since the data volume of the small file itself is small, a large amount of overhead is wasted on metadata operations and I / O addressings, and the proportion of time actually used for data reading is very small. Compared with reading a large file, reading small files with the same data volume may take dozens or even hundreds of times longer, seriously reducing the data processing efficiency. In addition, the traditional MapReduce task scheduling mechanism usually adopts a static allocation strategy and cannot dynamically adjust task allocation according to the real-time load conditions of cluster nodes. When dealing with a large number of small files, it is easy to cause uneven task allocation, overloading of some nodes, and low resource utilization, further exacerbating the performance bottleneck of small file processing.

[0005] To alleviate the small file problem, some small file merging solutions have emerged in the prior art, such as HadoopArchives (HAR), Sequence Files, CombineFileInputFormat, etc. These solutions can reduce the number of small files to a certain extent and relieve the pressure on the NameNode. However, these prior art solutions still have obvious defects:

[0006] Single and unoptimized merging strategy: Most of the existing small file merging solutions simply merge based on file size. For example, multiple small files are packed into an archive file or merged into a Sequence File. This merging method ignores important information such as file type and data correlation, resulting in limited efficiency improvement when reading the merged file subsequently, and may even reduce efficiency because the entire merged file needs to be scanned to find specific type of data. For example, when simply merging different types of log files, the efficiency is even lower when analyzing a specific type of log subsequently.

[0007] Lack of dynamics and flexibility: The existing merging operations are usually offline and batch, which need to be triggered manually or executed regularly. They cannot respond to newly generated small files in real time, nor can they dynamically adjust the merging strategy and parameters according to the actual business scenario and cluster status. This makes these solutions appear inadequate when dealing with rapidly growing and changing data.

[0008] Incomplete indexing and management mechanism: For the merged file, there is often a lack of a perfect mechanism on how to effectively index and manage it, and how to quickly locate and read the data of the original small files, resulting in an increase in the complexity of data access and management. For example, although the HAR archive file reduces the pressure on the NameNode, the efficiency of accessing a single file in the archive file is low and the entire archive file needs to be decompressed.

[0009] Lack of deep integration with new parallel computing frameworks: Although new parallel computing frameworks such as Spark show superior performance in processing big data, in the prior art, there is a lack of a solution to deeply integrate frameworks such as Spark with Hadoop and utilize their advantages for optimization for the problem of a large number of small files in Hadoop. Most of the existing solutions still rely on the traditional MapReduce framework and cannot give full play to the advantages of Spark's in-memory computing to accelerate the processing of small files.

[0010] The root cause of these defects is that most of the existing technical solutions are "repairs" based on the existing Hadoop architecture, lacking a systematic optimization idea to fundamentally solve the small file problem and failing to fully utilize emerging technical means to improve the small file processing efficiency. Therefore, there is an urgent need for a more efficient, intelligent, and flexible method for optimizing the reading of massive small files to meet the growing needs of big data applications. Summary of the Invention

[0011] The present invention provides a method, system, device, and medium for optimizing the reading of massive small files to solve the above problems.

[0012] The present invention provides a method for optimizing the reading of massive small files, including:

[0013] S1: Use the Spark computing engine to scan small files in the distributed file system to obtain the metadata information of the small files, where the metadata information includes file type, size, and file correlation information;

[0014] S2: Classify the small files based on the metadata information, and group the small files of the same type and having data correlation into one group;

[0015] S3: Real-time monitor the resource status of cluster nodes, including processor utilization rate, memory occupancy rate, and I / O load, calculate the load score of each node, and dynamically allocate small file task quotas;

[0016] S4: Use the Spark in-memory computing engine to perform the small file merging operation in parallel to generate a merged file containing multiple original small files;

[0017] S5: Create and maintain index information in an external database to record the location and attributes of each original small file in the merged file;

[0018] S6: Provide read and write interfaces through a dedicated service layer to locate and access specific small file content in the merged file.

[0019] In the above solution, in S1, first, utilize the distributed parallel processing ability of the Spark computing engine to efficiently scan massive small files in the Hadoop distributed file system; the introduction of the Spark engine can traverse and access the file metadata in HDFS more quickly compared to the traditional MapReduce framework; the scanned metadata information is crucial, which not only includes the traditional file size information but also innovatively covers file type and file correlation information; the file type information can be, for example, log files, transaction record files, image files, etc., and the file correlation information can be data correlation relationships based on dimensions such as timestamp, user ID, and business logic.

[0020] In S2, this step is one of the core innovations of the present invention, embodying a file type-driven merging strategy. Different from the traditional merging method based solely on file size, this step utilizes the metadata information obtained in S1, especially the file type and file correlation information, to conduct refined classification of small files. Only small files that meet both the conditions of being of the same type and having data correlation will be grouped together. For example, log files of the same type and belonging to the same time period or the same business process can be grouped together. This classification strategy is more in line with the actual application scenario and can ensure that the merged files have higher logical consistency and data locality during subsequent reading.

[0021] In S3, this step embodies a dynamic task scheduling mechanism. To overcome the deficiencies of the traditional static task scheduling in MapReduce, this step introduces a real-time monitoring mechanism to dynamically perceive the resource load conditions of each node in the cluster; the monitored resource status indicators include processor utilization rate, memory occupancy rate, and I / O load, and these indicators can comprehensively reflect the resource pressure and processing capacity of the nodes; based on these real-time monitoring data, the system calculates the load score of each node, and this score can comprehensively reflect the current load level of the node; then, according to the load scores of the nodes, small file task quotas are dynamically allocated, that is, more tasks are allocated to nodes with lower loads, fewer tasks are allocated to nodes with higher loads, and even task allocation is suspended.

[0022] In S4, this step is the core embodiment of the Spark-Hadoop fusion architecture; leveraging the in-memory computing advantage of the Spark computing engine, an efficient parallel merging operation is performed on the small files classified and grouped in S2; the Spark engine can load the data of the small files to be merged into memory for processing, reducing disk I / O operations and significantly improving the merging speed; parallel execution further utilizes the computing resources of the cluster, decomposing the merging task into multiple subtasks for parallel processing, further shortening the merging time; the result of the merging operation is to generate a merged file, which contains the content of multiple original small files, thus effectively reducing the number of small files in HDFS.

[0023] In S5, this step aims to solve the file access and management problems brought about by the merging of small files. To be able to quickly locate and access specific original small files in the merged file, the present invention creates and maintains index information in an external database, such as MongoDB, HBase, etc. The index information details the location (such as offset, data block number) and attributes (such as original file name, file size, file type, etc.) of each original small file in the merged file; the selection of the external database ensures the high availability and query efficiency of the index information. The maintenance of the index information includes the creation, update, and deletion of the index to ensure the accuracy and real-time nature of the index information.

[0024] In S6, this step is the ultimate manifestation of the optimization method, aiming to provide users with a convenient and efficient small file reading and writing service. By constructing a dedicated service layer, a unified reading and writing interface is provided externally. Users can use these interfaces to quickly locate the position of the original small file in the merged file according to the file name or other attributes of the original small file, and access its content. The service layer shields the complex merged file structure and indexing mechanism at the bottom layer. Users do not need to care about whether the small files are merged and the storage details after merging. They only need to access the small files like accessing ordinary files, which greatly simplifies the user operation and improves the user experience.

[0025] Preferably, the use of the Spark computing engine to scan small files in the distributed file system to obtain metadata information of the small files includes:

[0026] Configure the Spark computing engine to periodically scan the distributed file system to identify newly added small files;

[0027] Parse the file header information and / or content of the small file to extract file type, file size, and file correlation information;

[0028] Analyze the file name and / or file path to identify the data correlation between files, and the data correlation includes at least one of time correlation, user correlation, and business correlation.

[0029] In the above solution, the detailed steps of the metadata information acquisition process are specifically described, which include: First, configure the Spark computing engine to periodically scan the distributed file system, so as to be able to identify newly added small files in the distributed file system in a timely manner, ensuring the system's real-time processing ability for newly added small files; Second, by parsing the file header information and / or file content of the identified small files, key metadata information is extracted, and the metadata information at least includes file type, file size, and file correlation information, laying a data foundation for subsequent classification and merging of small files based on metadata information; Finally, by deeply analyzing the file name and / or file path, the potential data correlation between files is identified, and the data correlation specifically includes at least one of time correlation, user correlation, and business correlation, so as to provide a basis for subsequent merging of small files with strong correlation and improve the reading efficiency of the merged file.

[0030] Preferably, based on the metadata information, classifying small files and grouping small files of the same type and having data correlation includes:

[0031] Establish a file type classification rule library, and the file types include log files, transaction record files, medical image files, and user behavior data files;

[0032] According to the file type classification rule library and in combination with the extracted file type metadata, small files are classified into preset file type categories;

[0033] Based on the time range threshold of the file creation time, the matching degree of user identifiers, and the context relationship of the business process, it is determined whether small files within the same file type category have data relevance;

[0034] The small files of the same type with data relevance are sorted according to the strength of data relevance to form a group of small files to be merged.

[0035] In the above solution, the specific steps of classifying and grouping small files based on metadata information are described in detail, which include: First, establish a file type classification rule library, and pre-define common file types, such as log files, transaction record files, medical image files, and user behavior data files, etc., to provide standards and bases for classifying small files; Second, according to the file type classification rule library and in combination with file type metadata, small files are accurately classified into preset file type categories to achieve preliminary classification based on file types; Third, within the same file type category, based on multiple relevance judgment conditions such as the time range threshold of the file creation time, the matching degree of user identifiers, and the context relationship of the business process, it is comprehensively judged whether there is data relevance between small files to achieve further screening and relevance identification of small files of the same type; Finally, for small files of the same type determined to have data relevance, they are sorted according to the strength of data relevance to form the final group of small files to be merged, providing ordered data input for subsequent small file merging operations.

[0036] Preferably, the real-time monitoring of the resource status of cluster nodes, calculating the load scores of each node, and dynamically allocating small file task quotas include:

[0037] Deploy the Prometheus and Grafana monitoring platforms to collect the processor utilization rate, memory occupancy rate, and disk I / O load data of each node in the Hadoop cluster in real time;

[0038] Based on the preset weight coefficients, the collected processor utilization rate, memory occupancy rate, and disk I / O load data are weighted and summed to obtain the load scores of each node;

[0039] Dynamically update the load scores of each node, and the update period is at the second level;

[0040] Based on the load scores of each node, using a dynamic task allocation algorithm, calculate the number of Map tasks that can be allocated to each node, and the dynamic task allocation algorithm considers the total cluster task volume, node load scores, and node processing capabilities;

[0041] Using the Yarn resource manager, dynamically adjust the number of Map tasks for each node to match the task allocation with the node's load capacity;

[0042] When adjusting the number of Map tasks, consider the dependencies between tasks, prioritize the scheduling of dependent tasks, and ensure that the output results of dependent tasks are passed to subsequent tasks.

[0043] In the above solution, the specific implementation steps of dynamic task allocation are described in detail, including: First, deploy the Prometheus and Grafana monitoring platforms to collect real-time key resource status data of each node in the Hadoop cluster, including processor utilization, memory occupancy, and disk I / O load data, providing real-time data support for subsequent load score calculation and task allocation; Second, based on preset weight coefficients, perform weighted summation on the collected processor utilization, memory occupancy, and disk I / O load data to obtain the load score of each node, comprehensively reflecting the resource pressure and processing capacity of the node; Third, dynamically update the load score of each node with a second-level update period to ensure the real-time and accuracy of the load score, so as to timely reflect the changes in the resource status of cluster nodes; Fourth, based on the real-time load scores of each node, use a dynamic task allocation algorithm to calculate the number of Map tasks that can be allocated to each node. The dynamic task allocation algorithm comprehensively considers factors such as the total cluster task volume, node load score, and node processing capacity to achieve reasonable allocation of task quotas; Fifth, use the Yarn resource manager to dynamically adjust the number of Map tasks for each node to match the task allocation with the node's load capacity and achieve balanced utilization of cluster resources; Finally, when dynamically adjusting the number of Map tasks, fully consider the dependencies between tasks, prioritize the scheduling of dependent tasks, and ensure that the output results of dependent tasks can be correctly passed to subsequent tasks to ensure the correctness and efficiency of task execution.

[0044] Preferably, the use of the Spark in-memory computing engine to perform the small file merging operation in parallel to generate a merged file containing multiple original small files includes:

[0045] Taking the HDFS data block size as the unit, set the target size threshold for the merged file;

[0046] Adopt a pipeline merging method to parallelly read the small file data within the same small file group and splice and write the read data units into the target merged file;

[0047] The data unit includes a data unit header and stored data. The data unit header contains the data unit header length, data length, data offset, and file path information;

[0048] When the target merged file reaches the size threshold, create a new target merged file and continue to merge the remaining small files.

[0049] In the above solution, the specific implementation steps of the parallel small file merging operation are described in detail, which include: First, taking the HDFS data block size as the unit, set the target size threshold for the merged file. For example, the target size threshold can be set as an integer multiple of the HDFS data block size to ensure that the merged file can better adapt to the storage and management mechanism of HDFS. Second, adopt a pipelined merging method to parallelly read the small file data within the same small file group and splice and write the read data units into the target merged file, leveraging the parallel processing ability of the Spark in-memory computing engine to improve the efficiency of small file merging. Third, define the structure of the data unit, which includes two parts: the data unit header and the stored data. The data unit header contains key information such as the data unit header length, data length, data offset, and file path, which is used to describe the metadata of the data unit to ensure the parsability and traceability of the merged file. Finally, when the size of the target merged file reaches the pre-set target size threshold, dynamically create a new target merged file and continue to merge the remaining small files to achieve continuous merging and management of small files.

[0050] Preferably, create and maintain index information in the external database, and record the location and attributes of each original small file in the merged file, including:

[0051] Use the MongoDB database as the external database to construct a file index table and a data dictionary for the merged file;

[0052] The file index table records the file path, file size, permission attributes, owner, user group, last modification time, creation time, and merged file list information of the original file;

[0053] The data dictionary for the merged file records the file path, original file offset, data length, storage length, user group, last modification time, creation time, and merged file list information of the merged file;

[0054] Establish a file index with the original file path as the primary key and support multi-dimensional index queries based on file type, creation time, and data correlation.

[0055] In the above solution, the specific implementation solutions for creating and maintaining index information are described in detail, which include: First, the MongoDB database is used as an external database to store and manage index information. As a NoSQL database, MongoDB has characteristics such as high scalability, high performance, and flexible data models, and can meet the storage and query requirements of massive index information. Second, two data structures, namely the file index table and the merged file data dictionary, are constructed in the MongoDB database. The file index table is used to record the metadata information of the original files, and the merged file data dictionary is used to record the metadata information of the merged files, realizing a comprehensive index of the original files and the merged files. Third, the fields of the file index table and the merged file data dictionary are defined in detail. The file index table includes at least information such as the file path, file size, permission attributes, owner, user group, last modification time, creation time, and merged file list of the original file. The merged file data dictionary includes at least information such as the file path, original file offset, data length, storage length, user group, last modification time, creation time, and merged file list of the merged file, comprehensively recording the attribute and location information of the files. Finally, a file index is established with the original file path as the primary key, and multi-dimensional index queries based on file type, creation time, and data correlation are supported, improving the efficiency and flexibility of index queries and meeting the data access requirements in different scenarios.

[0056] Preferably, the specific small file content in the merged file is located and accessed by providing read and write interfaces through a dedicated service layer, including:

[0057] Construct a dedicated service layer based on the RPC protocol to provide read and write interfaces for file acquisition, file attribute modification, file content modification, and file deletion;

[0058] The read and write interfaces receive file operation requests from the client, and query the corresponding merged file and data offset information in the index information according to the requested file path;

[0059] Based on the queried index information, extract the data content of the target small file from the merged file and return it to the client through the RPC service.

[0060] In the above solution, the read and write interfaces provided by the dedicated service layer and their working processes are described in detail, which include: First, a dedicated service layer based on the RPC protocol is constructed to provide core read and write interfaces such as file acquisition, file attribute modification, file content modification, and file deletion. As an efficient remote call protocol, the RPC protocol can ensure efficient communication between the service layer and the client. Second, the read and write interfaces receive file operation requests from the client. The client identifies the target file by specifying the file path. After receiving the request, the service layer queries the corresponding merged file and data offset information in the index information according to the requested file path to achieve fast positioning of the target small file. Finally, based on the queried index information, the data content of the target small file is extracted from the merged file, and the extracted data content is returned to the client through the RPC service to complete the file reading operation. For operations such as file attribute modification, file content modification, and file deletion, corresponding operations are also performed based on the index information, and the operation results are returned to the client through the RPC service to achieve comprehensive management of the merged file.

[0061] Preferably, a system for optimizing the reading of a large number of small files includes:

[0062] A data acquisition module for scanning small files in a distributed file system using the Spark computing engine to obtain metadata information of the small files, where the metadata information includes file type, size, and file correlation information;

[0063] A file classification module for classifying small files based on the metadata information and grouping small files of the same type and having data correlation into one group;

[0064] A task allocation module for real-time monitoring of the resource status of cluster nodes, including processor utilization rate, memory occupancy rate, and I / O load, calculating the load score of each node, and dynamically allocating small file task quotas;

[0065] A file merging module for using the Spark in-memory computing engine to parallelly execute small file merging operations to generate a merged file containing multiple original small files;

[0066] An index recording module for creating and maintaining index information in an external database and recording the location and attributes of each original small file in the merged file;

[0067] A positioning and access module for positioning and accessing specific small file content in the merged file through the read and write interfaces provided by the dedicated service layer.

[0068] In the above solution, through the collaborative work of the data acquisition module, file classification module, task allocation module, file merging module, and location access module, the system can achieve efficient reading and processing of a large number of small files, reduce disk I / O operations, make full use of the advantages of in-memory computing, and meet the requirements of real-time services for rapid data processing. Through the dynamic task allocation mechanism of the task allocation module, resource allocation can be optimized, uneven load on Hadoop cluster nodes can be avoided, the utilization rate of cluster resources can be improved, and system energy consumption can be reduced. The system adopts a modular design, which is easy to expand and maintain, can adapt to the processing requirements of a large number of small files with different business types and different data scales, and provides strong technical support for the enterprise's big data strategy. By merging small files into large files through the file merging module and using the index record module for index management, the metadata management pressure on the NameNode can be reduced, system stability can be improved, and the risk of system crashes caused by excessive NameNode load can be reduced.

[0069] Preferably, a device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the above-mentioned method for optimizing the reading of a large number of small files is implemented.

[0070] Preferably, a computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned method for optimizing the reading of a large number of small files.

[0071] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0072] The present invention discloses a method, system, device, and medium for optimizing the reading of a large number of small files. By using the Spark computing engine to parallelly scan and merge small files, the metadata management burden on the NameNode is reduced, the disk I / O overhead is lowered, and by using the Spark computing engine, the data in the merged files can be quickly read, significantly improving the reading efficiency of small files; by merging small files, the number of metadata that the NameNode needs to manage is reduced, the memory consumption and CPU load of the NameNode are decreased, and the stability and scalability of the system are improved; through dynamic task allocation, the task allocation can be dynamically adjusted according to the real-time load conditions of the cluster nodes, avoiding node overload or resource idleness, and improving the overall processing efficiency; by creating and maintaining index information in an external database, the location and attributes of specific small files in the merged files can be quickly located, improving data access efficiency and traceability; through a flexible small file merging strategy, different types and related data can be adapted, improving the flexibility of the system and the business adaptation ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic flow diagram of an optimization method for reading a large number of small files provided by an embodiment of the present invention;

[0074] Figure 2 It is a schematic diagram of the fusion architecture of Spark and Hadoop provided by an embodiment of the present invention;

[0075] Figure 3 It is a schematic diagram of the module of an optimization system for reading a large number of small files provided by an embodiment of the present invention;

[0076] Figure 4 It is a schematic flow diagram of the Zookeeper service registration process provided by an embodiment of the present invention;

[0077] Figure 5 It is a schematic flow diagram of the file merging process provided by an embodiment of the present invention;

[0078] Figure 6 It is a schematic flow diagram of the file acquisition and processing process for the file being uploaded provided by an embodiment of the present invention;

[0079] Figure 7 It is a schematic flow diagram of the file acquisition and processing process provided by an embodiment of the present invention;

[0080] Figure 8 It is a schematic flow diagram of the file attribute modification and processing process for the file being uploaded provided by an embodiment of the present invention. Detailed implementation manners

[0081] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0082] As Figure 1 shown, the present application provides an optimization method for reading a large number of small files, including:

[0083] S1: Use the Spark computing engine to scan the small files in the distributed file system to obtain the metadata information of the small files, where the metadata information includes file type, size, and file correlation information;

[0084] S2: Classify the small files based on the metadata information, and group the small files of the same type and having data correlation into one group;

[0085] S3: Real-time monitor the resource status of the cluster nodes, including processor utilization rate, memory occupancy rate, and I / O load, calculate the load score of each node, and dynamically allocate the small file task quota;

[0086] S4: Use the Spark in-memory computing engine to perform the small file merging operation in parallel to generate a merged file containing multiple original small files;

[0087] S5: Create and maintain index information in an external database to record the location and attributes of each original small file in the merged file;

[0088] S6: Provide read and write interfaces through a dedicated service layer to locate and access the content of specific small files in the merged file.

[0089] Preferably, in step S1, the use of the Spark computing engine to scan small files in a distributed file system to obtain the metadata information of the small files includes:

[0090] Configure the Spark computing engine to periodically scan the distributed file system to identify newly added small files;

[0091] Parse the file header information and / or content of the small file to extract the file type, file size, and file correlation information;

[0092] Analyze the file name and / or file path to identify the data correlation between files, and the data correlation includes at least one of time correlation, user correlation, and business correlation.

[0093] In the above solution, through the combination of technical features such as periodic scanning and file header information / content parsing, the metadata information of small files can be obtained quickly and efficiently, reducing the system overhead and improving the overall processing efficiency; through multi-dimensional data parsing means, including the analysis of file header information, file content, file name, and file path, the metadata information of small files can be obtained comprehensively and accurately, providing reliable data support for subsequent small file classification and merging, and avoiding classification and merging errors caused by inaccurate metadata information; the technical feature of periodically scanning the distributed file system enables the system to timely discover and process newly added small files, ensuring the adaptability of the system to the dynamic data environment and improving the real-time performance and response speed of the system.

[0094] Preferably, in step S2, the classification of small files based on the metadata information and the grouping of small files of the same type and with data correlation into one group includes:

[0095] Establish a file type classification rule library, and the file types include log files, transaction record files, medical image files, and user behavior data files;

[0096] According to the file type classification rule library and in combination with the extracted file type metadata, divide the small files into preset file type categories;

[0097] Based on the time range threshold based on the file creation time, the matching degree of user identifiers, and the context relationship of business processes, determine whether small files within the same file type category have data relevance;

[0098] Sort the small files of the same type with data relevance according to the strength of data relevance to form a group of small files to be merged.

[0099] In the above solution, through the file type classification rule library and the multi-dimensional relevance judgment mechanism, it is possible to achieve refined classification and grouping of small files, effectively aggregate small files of the same type with high relevance, and provide high-quality data input for subsequent merging operations; the multi-dimensional relevance judgment mechanism can more accurately and comprehensively identify the data relevance between files, avoiding the problem of poor merging effect that may be caused by simple merging based only on file size or file name in traditional methods, and improving the accuracy and rationality of small file grouping; merging small files of the same type with high relevance together can improve the logical integrity and data locality of the merged file, so that when reading specific type or specific relevance data subsequently, there is no need to scan irrelevant content, significantly improving the reading efficiency.

[0100] Preferably, in step S3, the real-time monitoring of the resource status of cluster nodes, calculating the load score of each node, and dynamically allocating small file task quotas includes:

[0101] Deploy the Prometheus and Grafana monitoring platforms to collect the processor utilization rate, memory occupancy rate, and disk I / O load data of each node in the Hadoop cluster in real time;

[0102] Based on the preset weight coefficients, perform weighted summation on the collected processor utilization rate, memory occupancy rate, and disk I / O load data to obtain the load score of each node;

[0103] Dynamically update the load score of each node, and the update period is at the second level;

[0104] Based on the load score of each node, use a dynamic task allocation algorithm to calculate the number of Map tasks that can be allocated to each node, and the dynamic task allocation algorithm considers the total cluster task volume, node load score, and node processing ability;

[0105] Use the Yarn resource manager to dynamically adjust the number of Map tasks of each node to make the task allocation match the node load capacity;

[0106] When adjusting the number of Map tasks, consider the dependency relationship between tasks, give priority to scheduling dependent tasks, and ensure that the output results of dependent tasks are passed to subsequent tasks.

[0107] In the above solution, by monitoring the resource status of cluster nodes in real time and dynamically adjusting the number of Map tasks according to the load score, the load of cluster nodes can be effectively balanced, avoiding the situation where some nodes are overloaded and some nodes are idle, and improving the balanced utilization rate of cluster resources; the dynamic task allocation mechanism can reasonably allocate task quotas according to the actual processing capabilities of cluster nodes, enabling more full utilization of cluster resources, reducing resource waste, and improving resource utilization efficiency; by dynamically adjusting the number of Map tasks to match the task allocation with the node load capacity, it is possible to avoid the situation of low task execution efficiency or failure caused by node overload, improve task processing efficiency, and shorten task execution time; the load-balanced task allocation strategy can reduce the risk of system crashes caused by node overload, improve the stability and reliability of the cluster, and ensure the smooth completion of massive small file processing tasks.

[0108] In an embodiment provided by the present application, an intelligent Map task dynamic adjustment system is designed. The core of this system lies in real-time monitoring of the load conditions of each node in the Hadoop cluster, and the monitoring indicators cover key resource indicators such as CPU usage rate, memory occupancy rate, and disk I / O busy degree. Its working principle is based on the real-time perception and dynamic analysis of cluster resources. By building a Prometheus and Grafana monitoring platform on Hadoop, the resource usage conditions of each node in the cluster can be accurately obtained. Prometheus is responsible for collecting and storing the resource data of nodes, while Grafana is used to visually display this data, facilitating administrators to intuitively understand the cluster status.

[0109] With the help of the Prometheus and Grafana monitoring platforms, the index data such as CPU usage rate, memory occupancy rate, and disk I / O of each node in the cluster are collected in real time. These data are updated at a second-level frequency to ensure that the system can timely obtain the latest node resource status.

[0110] According to the real-time resource usage conditions obtained by the monitoring platform, a preset algorithm is used to dynamically calculate the number of Map tasks suitable for each node to undertake. This algorithm comprehensively considers various resource indicators of the node and calculates a resource score for each node to reflect its current processing ability.

[0111] Suppose the CPU usage rate of node i is CPU i , the memory usage rate is Memory i , the disk I / O utilization rate is IO i , and the weights of CPU, memory, and disk I / O are W cpu , W Memory , W IO , and the maximum resource utilization rate is 100%. Then the calculation formula for the load value of node i is:

[0112]

[0113] Calculate the number of Map tasks to be allocated to each node according to the load score of each node.

[0114] Suppose there are a total of S Map tasks to be allocated to N nodes, and the task quota Quota of node i i The calculation formula is:

[0115]

[0116] Use the Yarn resource manager to dynamically adjust the number of Map tasks. If it is found that the resource utilization rate of some nodes is low, such as the CPU usage rate is lower than 30% and the memory occupancy rate is lower than 40%, then the number of Map tasks on these nodes can be increased; on the contrary, if the resources of some nodes are approaching saturation, such as the CPU usage rate is higher than 80% or the memory occupancy rate is higher than 90%, then the number of Map tasks on these nodes is reduced.

[0117] When adjusting the number of Map tasks, fully consider the sequentiality and dependency of task execution. For example, for tasks A and B with data dependencies, if task B depends on the output result of task A, then when adjusting the number of tasks, ensure that task A is executed before task B, and the execution result of task A can be correctly passed to task B.

[0118] Compared with the traditional task allocation algorithm, the preset algorithm of this application has significant advantages. Traditional algorithms often rely on static resource allocation strategies and cannot be dynamically adjusted according to the real-time load conditions of nodes. However, this algorithm can perceive the changes in cluster resources in real time and achieve precise task allocation. By introducing multi-dimensional resource metrics (CPU, memory, disk I / O) and reasonable weight settings, it can evaluate the processing capabilities of nodes more comprehensively and accurately. For example, in a compute-intensive business scenario, the CPU weight can be appropriately increased so that nodes with strong computing capabilities can undertake more tasks; in an I / O-intensive business scenario, the disk I / O weight is increased to ensure that nodes with good I / O performance can efficiently process tasks. This dynamic and precise task allocation method can effectively improve the utilization rate of cluster resources, reduce task execution time, and enhance the overall performance of the system.

[0119] Preferably, in step S4, the use of the Spark in-memory computing engine to parallelize the small file merging operation to generate a merged file containing multiple original small files includes:

[0120] Set the target size threshold of the merged file in units of the HDFS data block size;

[0121] Adopt a pipelined merging method to read the data of small files in the same small file group in parallel, and splice and write the read data units into the target merged file;

[0122] The data unit includes a data unit header and stored data, and the data unit header contains the data unit header length, data length, data offset, and file path information;

[0123] When the target merged file reaches the size threshold, create a new target merged file and continue to merge the remaining small files.

[0124] In the above solution, by utilizing the parallel processing ability of the Spark memory computing engine and the pipelined merging method, the efficiency of small file merging can be significantly improved, the merging time can be shortened, and the system overhead can be reduced; by defining the data unit structure and splicing and writing the data units into the target merged file, a structured merged file can be generated, which is convenient for subsequent index and read services to perform data parsing and access; setting the target size threshold of the merged file in units of the HDFS data block size enables the merged file to better adapt to the HDFS storage and management mechanism and improve data access efficiency; the mechanism of dynamically creating a new target merged file ensures the continuity and scalability of the small file merging operation and can adapt to the continuously growing small file data.

[0125] Preferably, in step S5, creating and maintaining index information in the external database, and recording the location and attributes of each original small file in the merged file include:

[0126] Use the MongoDB database as the external database to construct a file index table and a data dictionary for the merged file;

[0127] The file index table records the file path, file size, permission attributes, owner, user group, last modification time, creation time, and merged file list information of the original file;

[0128] The data dictionary for the merged file records the file path, original file offset, data length, storage length, user group, last modification time, creation time, and merged file list information of the merged file;

[0129] Establish a file index with the original file path as the primary key, and support multi-dimensional index queries based on file type, creation time, and data relevance.

[0130] In the above solution, by constructing a file index table and a merged file data dictionary and storing and managing them using the MongoDB database, effective indexing and management of the merged files can be achieved, reducing the complexity of data management; establishing a file index with the original file path as the primary key and supporting multi-dimensional index queries can improve the efficiency and flexibility of index queries, meet the data access requirements in different scenarios, and shorten the data query time; the complete index information can quickly locate the position and attributes of the original small files in the merged files, providing guarantee for subsequent fast data access and improving the data reading efficiency; using the MongoDB database as an external database has characteristics such as high scalability, high performance, and high availability, which can enhance the scalability and reliability of the system and meet the storage and query requirements of massive index information.

[0131] Preferably, in step S6, the providing read and write interfaces through the dedicated service layer to locate and access the content of specific small files in the merged file includes:

[0132] Construct a dedicated service layer based on the RPC protocol to provide read and write interfaces for file acquisition, file attribute modification, file content modification, and file deletion;

[0133] The read and write interfaces receive file operation requests from the client, and query the corresponding merged file and data offset information in the index information according to the requested file path;

[0134] Based on the queried index information, extract the data content of the target small file from the merged file and return it to the client through the RPC service.

[0135] In the above solution, by constructing a dedicated service layer based on the RPC protocol and providing read and write interfaces such as file acquisition, file attribute modification, file content modification, and file deletion, a convenient and user-friendly data access and management service can be provided for users, reducing the user's usage threshold; the technical feature of querying the merged file and data offset information based on the index information can quickly locate and access the content of specific small files in the merged file, avoiding the inefficient method of full file scanning and improving the data reading efficiency; the dedicated service layer can provide efficient data access and management services, shorten the data access time, enhance the user experience, and meet the user's fast access requirements for massive small file data; by constructing the dedicated service layer, the data access and management logic is decoupled from the underlying storage and computing logic, enhancing the maintainability and scalability of the system and facilitating subsequent function expansion and system upgrade.

[0136] In one embodiment provided in the present application, a set of tightly coupled Spark-Hadoop hybrid architecture is built to achieve deep integration of the two in terms of data storage, task scheduling, and computing resource sharing. In terms of data storage, the Spark distributed computing platform framework is introduced into Hadoop. Through the real-time computing engine in Spark, the files in HDFS are scanned in real time, small files of the same type are merged into large data files, and the file index information is recorded in MongoDB, thereby reducing the memory pressure of the NameNode node, thereby improving the operation stability of HDFS. In terms of task scheduling, Spark's task scheduler works with Hadoop's Yarn resource manager to reasonably allocate task resources according to the priority, resource requirements, and node load of the task. In terms of computing resource sharing, Spark and Hadoop can share CPU, memory, disk and other resources in the cluster to improve resource utilization. For example, Spark can use idle memory resources in the Hadoop cluster for data caching and calculation, and Hadoop can also use Spark's efficient computing power to handle complex data analysis tasks.

[0137] Merge according to file type, abandoning the existing algorithm of merging small files into large files, which only considers the size of the file, but not the characteristics of the file itself, resulting in low efficiency when reading the merged file. Merge small files based on file type. On the one hand, the pipeline merge method can improve the merge efficiency. On the other hand, the simple merge algorithm is conducive to improving the reading efficiency of the subsequent small files.

[0138] After the files on HDFS are merged, the file information needs to be re-indexed. The file information index designed by the present invention not only ensures that the files can be correctly indexed, but also records information such as application access on Spark. A file index table is established in the database MongoDB, and the table structure includes fields such as file path (as primary key), file size, permission attributes, owner, user group, last modification time, creation time, and merged file list. Compared with the traditional file indexing method, this index design adds a merged file list field, which can record in detail the distribution information of the original file in different merged files after merging. For example, an original file may be merged into multiple different large files, and the information such as its offset and data length in each merged file can be accurately recorded through the merged file list field, which is convenient for subsequent file reading and data recovery.

[0139] After the files are merged, they can no longer be obtained through normal means, so a set of reliable file reading and writing services is provided for reading and writing files.

[0140] like Figure 2As shown in the figure, this architecture designs a real-time HDFS file discovery and merging service based on the Spark real-time computing platform. Through this service, all small files of the same type on HDFS can be merged according to the HDFS data block size, thus reducing the sharp increase in the memory consumption of the NameNode caused by the excessive number of files. After the files on HDFS are merged, the file information is re-indexed. The new index not only needs to ensure that the files can be correctly indexed, but also needs to record information such as access by applications on Spark, so as to achieve more operations based on file information. After the files are merged, a reliable file read and write service is provided for operations such as reading and writing files.

[0141] Preferably, as Figure 3 shown, a system for optimizing the reading of a large number of small files includes:

[0142] A data acquisition module, which is used to scan small files in a distributed file system using a Spark computing engine to obtain metadata information of the small files, and the metadata information includes file type, size, and file association information;

[0143] A file classification module, which is used to classify small files based on the metadata information, and group small files of the same type and with data association into one group;

[0144] A task allocation module, which is used to monitor the resource status of cluster nodes in real time, including processor utilization rate, memory occupancy rate, and I / O load, calculate the load score of each node, and dynamically allocate small file task quotas;

[0145] A file merging module, which is used to use the Spark in-memory computing engine to parallelize the small file merging operation to generate a merged file containing multiple original small files;

[0146] An index recording module, which is used to create and maintain index information in an external database and record the location and attributes of each original small file in the merged file;

[0147] A location access module, which is used to provide read and write interfaces through a dedicated service layer to locate and access specific small file contents in the merged file.

[0148] In the above solution, the present invention aims to comprehensively optimize the process of reading a large number of small files based on Hadoop from multiple dimensions. By innovatively combining enhanced MapReduce technology with the introduction of a new parallel processing framework Spark, and supporting it with a unique task scheduling strategy, the goal is to improve the efficiency of reading and processing a large number of small files, reduce system resource consumption, and enhance the adaptability of the Hadoop ecosystem in complex data scenarios. Specifically, the enhanced MapReduce technology will reasonably allocate tasks according to the real-time load conditions of cluster nodes through a dynamic task allocation mechanism; the introduction of the Spark framework takes advantage of its memory-based computing model to merge and quickly read relevant small files; and the task scheduling strategy will ensure that the entire system can operate efficiently and stably when processing a large number of small files.

[0149] In an embodiment provided by the present application, the monitoring platform is built and the MapReduce module is initialized: Prometheus and Grafana monitoring platforms are built on Hadoop to initialize the enhanced MapReduce module. The node load monitoring frequency is set to 10 seconds to ensure that the resource usage of each node in the cluster can be obtained in a timely manner. By configuring the data source of Prometheus and the dashboard of Grafana, real-time monitoring and visual display of indicators such as CPU usage rate, memory occupancy rate, and disk I / O busy degree are realized. At the same time, the parameters of the enhanced MapReduce module are initialized, including weight parameters in the preset algorithm, initial thresholds for task allocation, etc.

[0150] Installation and configuration of the Spark cluster: On the basis of the existing Hadoop cluster, install and configure the Spark cluster. Ensure that the communication link between the Spark cluster and the Hadoop cluster is unobstructed. By configuring network parameters and communication protocols, smooth data interaction is achieved. For example, set the communication port between the Master node of Spark and the NameNode node of Hadoop, and configure the data transfer protocol between the Executor node of Spark and the DataNode node of Hadoop. At the same time, build an interface module between the two so that Spark can directly access the data on HDFS, and Hadoop can also call the computing power of Spark.

[0151] MongoDB Database Cluster Installation and Configuration: This cluster is used to record information about files on HDFS, such as the original file path, original file size, file permission information, as well as information such as the merged file path, file size, and offset of the file in the merged block. At the same time, it can record some information when Spark applications operate on the file. When users extract files, they can find the basic information of the files to be extracted through the file information recorded in the MongoDB database. To avoid the situation where the information of the files being merged is not recorded in MongoDB, when the file information cannot be queried in the MongoDB database, the files need to be queried from the original file path according to the provided information. When users need to append files, they need to first obtain the merged information of the original files, create a merged data block of the information, write the content of the file to be appended into the new merged block, and finally write the content of the data block to the MongoDB database to meet the file append requirement. When users need to delete files, they only need to transfer the recorded file information specified in MongoDB to the recycle bin table. The system can periodically re-merge the content of the merged data blocks in the recycle bin table to achieve file deletion. At the same time, data recovery can also be achieved based on this table. When a Spark application executes a certain task, some information about the task can be recorded in the MongoDB database. In this way, when multiple tasks are executed in parallel or jointly, the purpose of sharing file information and task information can be achieved, and more complex business requirements can be realized.

[0152] Zookeeper Installation and Configuration: Install and configure Zookeeper. As a distributed coordination service, Zookeeper can forward different requests to different response servers for processing under high concurrency. After all file operation service nodes register with Zookeeper, information sharing between each node can be achieved through the Watch mechanism of Zookeeper.

[0153] such as Figure 4As shown in the figure, each service node creates a service root node merge on the Zookeeper root node. This node is a persistent node and is used to mount all child nodes of the file merging service. When the child nodes under the merge node change, Zookeeper will notify each node in a timely manner. The root node usually only needs to be created once. However, considering that Zookeeper will periodically clean up unused nodes, if all service nodes fail to start or Zookeeper is restarted, the root node will be lost. Moreover, directly creating the root node only requires accessing Zookeeper once, while creating the root node after judging whether it exists requires accessing Zookeeper at least once. At the same time, considering that the role of the root node is to mount child nodes and the root node itself does not record any data information, the root node is directly created here without the need to judge in advance whether the root node exists. Then, each service node registers its own child node information under the merge node, uses its own host IP and port as the node name, and writes the current node status value into the child node. When the number of service instances changes, Zookeeper will notify all nodes registered under merge in a timely manner. Among them, the status value written into the child node is the data to be shared through Zookeeper. Each data is directly separated by a semicolon and finally concatenated into a complete string and saved in the Zookeeper child node.

[0154] Unified scheduling center startup and initialization: Start the unified scheduling center and load the preset task scheduling algorithm parameters, resource allocation models, and priority rules, etc. The preset task scheduling algorithm parameters include weight parameters in node load score calculation, adjustment thresholds for task allocation, etc.; the resource allocation model is designed according to different business scenarios and data characteristics. For example, for businesses with high real-time requirements, resources are preferentially allocated to nodes that process such tasks; the priority rules determine the execution priority of tasks based on factors such as the urgency of tasks and the amount of data. Initialize the task request queue to store task requests to be processed.

[0155] File opening and status acquisition: After filtering out files of the same type, when the total size of the small files to be merged reaches 128MB, the files can start to be merged. Spark obtains the path of a file, opens the file, and obtains file status information, including file byte count, file name, file control permissions, and other information. The opening and status acquisition of the file are implemented through the file operation API of Hadoop to ensure the security and stability of the operation.

[0156] Calculation of mergeable bytes: Based on the association among the file bytes, file path, block size set by HDFS, and the size of the currently merged file in the file status information, calculate the bytes that can be merged in this loop. The mergeable bytes need to satisfy the following relationship:

[0157] L head = 22 + L path , L write = Min(L left - L head , L file + L head ).

[0158] Among them, L path is the number of bytes occupied by the file path, L head is the number of bytes occupied by the data unit header, is the remaining mergeable bytes of the currently merged file, is the number of file bytes, is the number of bytes that can be written this time, that is, the total number of mergeable bytes. By accurately calculating the mergeable bytes, problems such as data overflow and overly large merged files are avoided, improving the merge efficiency and the standardization of file management.

[0159] Data reading and unit splicing: After obtaining the mergeable bytes, read L write bytes from the current position of the file. Splice the read data content with the data unit header to obtain a complete data unit. The data unit header includes the data unit header length (2 bytes, recording the total number of bytes after the file path), data length (4 bytes, recording the data length of the current unit), data offset (4 bytes, recording the offset of the data in the original file), and file path (N bytes, recording the absolute path of the original file). By reasonably designing the data unit header structure, the data integrity and traceability of the merged file are ensured.

[0160] Data writing and loop processing: Write the data unit into the target file. If the original file has not been completely merged, continue to execute the above steps until the original file is merged. When L left ≤ L head , a new merged file needs to be created to store the new data. Through this loop processing method, the merge operation of multiple small files can be efficiently completed, reducing the number of files and the load on the NameNode. Each data unit in a merged file is independent of each other, and the number of data units in a merged file depends on the actual number of merged files and the data block size defined by the HDFS distributed file system. The file merge process is as Figure 5 shown.

[0161] A file to be merged is regarded as a data unit, while a merged file is composed of multiple data units. Each data unit records all the information of the file it represents, including the file path, file size, etc., and also records the information of the current data unit.

[0162]

[0163] Among them, the data unit header length, data length, data offset, and file path constitute the data unit header. The data unit header length records the total length of the data unit header. The data length is used to record the total number of bytes of the data stored in the current data unit. The data offset is used to record the position of the data stored in the current data unit in the original file before merging, and this position is the offset based on the 0th byte. The file path records the absolute path where the original file is located, and this path should uniquely identify a file.

[0164] When the same file is written by multiple threads, it will cause the file content to be disordered and the data to be chaotic. Therefore, when performing file merging, it is not allowed for each partition to write the same data to the same file, but instead, it is necessary to write to multiple different files. Therefore, a file distribution algorithm is designed to distribute files to different tasks for processing according to specific rules, while ensuring that the same file is not processed in different task units. The file distribution algorithm is implemented by triggering a listener at the start of the Task listener provided by Spark. In this listener, the ID of the current Task can be obtained, and this ID is the unique identifier of the task. After taking the modulo of the ID, the obtained value can be used as the file identifier. As long as the modulo is not less than the maximum value of the number of file processing times, it can be ensured that the files processed by each Task will be written to different files, and the situation where multiple Tasks write to the same file simultaneously will not occur.

[0165] The specific algorithm process of file distribution is as follows: First, obtain the TaskId and take the modulo of the TaskId (the modulo value is equal to the number of files), and then obtain the file from the file list through the modulo result.

[0166] Index database design: HDFS uses the NameNode node to record the file index information in HDFS and the index information of each data block. After the file is merged, the original file will be deleted to release storage space. Therefore, it is necessary to record and establish an index for the merged file and the relationship between the original file and the merged file. Store the index in the MongoDB database. The data dictionary of the newly created HDFS file index in the MongoDB database is shown in the following table.

[0167]

[0168] The file path serves as the primary key to uniquely identify the file; fields such as file size, permission attributes, owner, user group, last modification time, and creation time record the basic attributes of the file; the merged file list records the distribution information of the original files in different merged files after merging.

[0169] Design of the data dictionary for the merged file: The data dictionary of the merged file is shown in the following table.

[0170]

[0171]

[0172] The file path is used to retrieve the location of the merged file in the HDFS file system; the original file offset records the offset of the stored original file data in the merged file; the data length records the length of the stored original file data; the storage length records the number of bytes occupied by the original file data after merging in the merged file; the compression type records the way of compressing the stored data, such as GZ compression, etc. By reasonably designing the data dictionary of the merged file, it facilitates the subsequent reading and data parsing of the merged file.

[0173] Use system commands to upload local files to the HDFS file system. Enter the host background and use relevant commands to upload the prepared files to the HDFS file system. The program will scan the queue to be merged in real time and merge all the files in the queue. After completing the upload and merging, check the results of the real-time file merging. Use the MongoDB shell command line method to connect to the MongoDB database and query the uploaded file index.

[0174] From the start of file upload to the HDFS file system, after the file lands in the HDFS file system to complete the file upload work, to the final completion of file merging and removal of the original file content.

[0175] During the entire cycle, the file is in the following three stages: file uploading, file landing, and file merging completed.

[0176] During these three stages, there may be requests to obtain the file content externally, and the required response methods are different for files in different processing stages.

[0177] Stage 1, file uploading:

[0178] In a traditional file system, when a file is being written, it is possible to determine whether the file has completed the write operation by observing changes in the file size and the file modification time. Similarly, in the HDFS file system, when a file is in the process of being uploaded, its file size and file modification time are constantly changing. However, due to factors such as caching and network, these changes are not reflected in real time. Therefore, it is necessary to check the changes in the file size and file modification time at multiple time points to determine whether the file has been uploaded successfully.

[0179] When it is determined that a file is in the process of being uploaded, it is certain that the file has not started to be merged. Therefore, it is only necessary to extract the file from the original HDFS and return it to the caller. However, since the file is being written, the caller should only have read-only permissions and cannot perform operations on the file.

[0180] The process of obtaining a file in the first stage is as Figure 6 shown.

[0181] Stage Two: File Landing:

[0182] When a file can be retrieved in the HDFS file system and is not in the process of being uploaded, then the file is in the landed state.

[0183] Similar to the file being uploaded, for a landed file, it is only necessary to extract the file from the original HDFS and return it to the caller.

[0184] At the same time, since the file may have already started, or is about to start, the merging process, to avoid data inconsistency between the merged data and the original data, a file in the landed state also only has read permissions and cannot perform other types of operations on the file.

[0185] Stage Three: File Merging Completed:

[0186] When a file can be retrieved in the file index, it indicates that the file has undergone the merging process. At this time, it is necessary to obtain the merging information from the index, and after extracting the data from the merged file according to the merging information, integrate the data and return it to the caller. The process of obtaining a file in the third stage is as Figure 7 shown.

[0187] File Attribute Modification Process

[0188] File attributes include the file owner, the group to which the file owner belongs, the operation permissions of the file owner on the file, the operation permissions of the group to which the file owner belongs on the file, and the operation permissions of other groups on the file. Among them, the operation permissions on the file can be modified.

[0189] The file operation permissions include read-only, write-only, and executable. Each type is represented by one byte. When a file has the operation permission, the byte corresponding to the permission is displayed with the corresponding character, otherwise the character "-" is used as a placeholder. Read-only is marked with "r", write-only is marked with "w", and executable is marked with "w".

[0190] After the merge, the file's permission information has been written into the MongoDB file index. Therefore, when you need to change the file attributes, you only need to overwrite the new file attributes to the existing file attribute fields. The execution process of modifying file attributes is as follows: Figure 8 shown.

[0191] File content modification

[0192] Different from traditional file systems, the HDFS file system only allows files to be appended and does not support random file modifications. Therefore, file content modification is also aimed at appending data to existing files.

[0193] When the caller needs to append to a file, it still needs to obtain the file information first and determine the current status of the file.

[0194] 1) When a file is being uploaded or has landed, writing to the file is not allowed. In this case, an error message needs to be returned to the caller to indicate that writing to the file is not allowed.

[0195] 2) When the file is in the process of being merged, the caller is allowed to append content to the file. The specific process is as follows:

[0196] a) Obtain the data content of the original file;

[0197] b) Write the data to be appended into a temporary HDFS file first;

[0198] c) Merge the temporary file according to the normal merging process

[0199] d) After the merge is completed, the file list information in the new merge information is appended to the index of the original file;

[0200] e) Integrate the data content of the original file and the newly appended data content into an HDFS file, and calculate its file length and other information;

[0201] f) Overwriting the information in the original file index with the new file information;

[0202] g) Delete the HDFS temporary file with additional information;

[0203] h) At this point, data appending is completed.

[0204] 3) When no file information is found, it can be merged into a single file. The specific process is as follows:

[0205] a) Write the data to be appended to an HDFS file whose path is the same as the path to which the data is to be appended.

[0206] The specified file path is consistent;

[0207] b) Merge the new file according to the normal merging process;

[0208] c) The merge is completed, that is, the data append is completed when the file information is not found.

[0209] File Deletion

[0210] When deleting a specified file, you still need to determine the status of the file first and perform specific operations based on the file status.

[0211] 1) When a file is being uploaded or has been landed, it is not allowed to delete the file. At this time, an error message needs to be returned to the caller to indicate that the file cannot be deleted.

[0212] 2) When the file is merged, the caller is allowed to delete the file data. The specific process is as follows:

[0213] a) Get the index of the file;

[0214] b) Obtain all merged files based on the information in the file index;

[0215] c) according to the information in the file index, the merged file is divided into two parts, one is the data before the data to be deleted, and the other is the data after the data to be deleted;

[0216] d) Rejoin the two parts of data;

[0217] e) traverse the contents of each data block in the following data, find its corresponding file index, and change the offset in the merged file information list in the index to the current offset minus the length of the data block to be deleted;

[0218] f) Delete the index information of the file to be deleted;

[0219] g) At this point, the file is deleted.

[0220] When the file information is not found, an error message is returned, indicating that the file does not exist.

[0221] External Services

[0222] RPC (Remote Procedure Call) service is a remote procedure call, which can provide a remote function call service for the client. Through the RPC service, the client developer can call the function provided by the remote server just like calling a normal function locally, so as to achieve business processing. RPC service is a common system decoupling method. This external service adopts the RPC remote procedure call method.

[0223] A complete RPC service interaction is divided into a server and a client. The server is used to receive remote call requests from the client. In this design, the server provides the ability to obtain merged files, modify file attributes, modify file content, and delete files. In the RPC server, external access requests are captured by listening to the specified port. After verifying the legitimacy of user information and commands, these requests are processed concurrently using multiple threads, and finally the processing results are returned to the calling client, thereby completing an RPC request.

[0224] The server first needs to define the request message format and return message format required for RPC access. The format of the request message is shown in the following table.

[0225]

[0226]

[0227] The return code and the result are recorded in the return message. The returned result has different meanings in different file operation processes. For example, in the file acquisition process, the path of the extracted file is written in the return result. The server needs to encapsulate all required requests into a class and register this class on the RPC server. Based on the capabilities provided by the file operation service, the server defines the following four file operation requests:

[0228] Get: Get a file. Through this interface, the remote client can get the specific content of the file. When the client initiates a request, it writes the path of the file to be obtained in Path. After the server obtains the file content according to the file path, it forwards the file content to the client.

[0229] Chmod: File attribute modification. Through this interface, the remote client can modify file attributes.

[0230] Modify: file content modification. Through this interface, the remote client can append new data content to the original file.

[0231] Delete: File deletion. Through this interface, the remote client can delete files.

[0232] Regular health check mechanism: Regularly (e.g., weekly) conduct health checks on the Hadoop cluster, Spark cluster, and interface module. The check content includes software vulnerability detection, hardware fault potential troubleshooting, etc. Use professional vulnerability scanning tools to detect the software versions of Hadoop and Spark, and promptly discover and fix possible security vulnerabilities. For hardware devices, such as the CPU, memory, disk, etc. of the server, conduct performance monitoring and fault prediction, and replace the hardware devices with potential faults in advance to ensure the stable operation of the system.

[0233] Dynamic optimization and adjustment strategy: According to the system operation logs, performance monitoring data, and business feedback, dynamically optimize and adjust the algorithm parameters of the unified scheduling center, task priority rules, and processing strategies of each cluster. For example, by analyzing the system operation logs and finding that the execution efficiency of certain tasks is low during a specific time period, the task priority rules can be adjusted to schedule these tasks during the time period when the system load is low.

[0234] The above are the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. An optimization method for reading a large number of small files, characterized in that, Including: Using the Spark computing engine to scan small files in a distributed file system to obtain metadata information of the small files, where the metadata information includes file type, size, and file correlation information; Classifying the small files based on the metadata information, and grouping small files of the same type and having data correlation into one group; Real-time monitoring the resource status of cluster nodes, including processor utilization rate, memory occupancy rate, and I / O load, calculating the load score of each node, and dynamically allocating small file task quotas; Using the Spark in-memory computing engine to parallelly execute the small file merging operation to generate a merged file containing multiple original small files; Creating and maintaining index information in an external database to record the location and attributes of each original small file in the merged file; Providing read and write interfaces through a dedicated service layer to locate and access specific small file content in the merged file.

2. The optimized method for reading a large number of small files according to claim 1, wherein The using the Spark computing engine to scan small files in a distributed file system to obtain metadata information of the small files includes: Configuring the Spark computing engine to periodically scan the distributed file system to identify newly added small files; Parsing the file header information and / or content of the small files to extract file type, file size, and file correlation information; Analyzing file names and / or file paths to identify data correlation between files, where the data correlation includes at least one of time correlation, user correlation, and business correlation.

3. The optimized method for reading a large number of small files according to claim 2, characterized in that, The classifying the small files based on the metadata information, and grouping small files of the same type and having data correlation into one group includes: Establishing a file type classification rule library, where the file types include log files, transaction record files, medical image files, and user behavior data files; According to the file type classification rule library, combining the extracted file type metadata to divide the small files into preset file type categories; Based on the time range threshold of file creation time, the matching degree of user identifiers, and the context relationship of business processes, determining whether small files within the same file type category have data correlation; Sorting the small files of the same type with data correlation according to the strength of data correlation to form a grouping of small files to be merged.

4. The optimized method for reading a large number of small files according to claim 3, characterized in that The real-time monitoring the resource status of cluster nodes, calculating the load score of each node, and dynamically allocating small file task quotas includes: Deploying the Prometheus and Grafana monitoring platforms to collect processor utilization rate, memory occupancy rate, and disk I / O load data of each node in the Hadoop cluster in real time; Based on preset weight coefficients, performing weighted summation on the collected processor utilization rate, memory occupancy rate, and disk I / O load data to obtain the load score of each node; Dynamically updating the load score of each node, and the update period is at the second level; Based on the load score of each node, using a dynamic task allocation algorithm to calculate the number of Map tasks that can be allocated to each node, where the dynamic task allocation algorithm considers the total cluster task volume, node load score, and node processing capacity; Using the Yarn resource manager to dynamically adjust the number of Map tasks of each node to make the task allocation match the node load capacity; When adjusting the number of Map tasks, consider the dependencies between tasks, prioritize scheduling dependent tasks, and ensure that the output results of dependent tasks are passed to subsequent tasks.

5. The optimization method for reading a large number of small files according to claim 4, wherein The parallel execution of small file merging operations using the Spark in-memory computing engine to generate a merged file containing multiple original small files includes: Set the target size threshold for the merged file in units of the HDFS data block size; Adopt a pipelined merging method to parallelly read the small file data within the same small file group and splice and write the read data units into the target merged file; The data unit includes a data unit header and stored data, and the data unit header contains the data unit header length, data length, data offset, and file path information; When the target merged file reaches the size threshold, create a new target merged file and continue to merge the remaining small files.

6. The optimized method for reading a large number of small files according to claim 5, characterized in that, The creation and maintenance of index information in an external database to record the location and attributes of each original small file in the merged file includes: Use the MongoDB database as the external database to build a file index table and a merged file data dictionary; The file index table records the file path, file size, permission attributes, owner, user group, last modification time, creation time, and merged file list information of the original file; The merged file data dictionary records the file path, original file offset, data length, storage length, user group, last modification time, creation time, and merged file list information of the merged file; Establish a file index with the original file path as the primary key and support multi-dimensional index queries based on file type, creation time, and data relevance.

7. A method for optimizing the reading of a large number of small files according to claim 6, characterized in that, The provision of read and write interfaces through a dedicated service layer to locate and access specific small file content in the merged file includes: Build a dedicated service layer based on the RPC protocol to provide read and write interfaces for file acquisition, file attribute modification, file content modification, and file deletion; The read and write interfaces receive file operation requests from the client, query the corresponding merged file and data offset information in the index information according to the requested file path; Based on the queried index information, extract the data content of the target small file from the merged file and return it to the client through the RPC service.

8. An optimization system for reading a large number of small files, characterized in that, Includes: A data acquisition module for scanning small files in a distributed file system using the Spark computing engine to obtain metadata information of the small files, where the metadata information includes file type, size, and file relevance information; A file classification module for classifying small files based on the metadata information and grouping small files of the same type and with data relevance into one group; A task allocation module for real-time monitoring of the resource status of cluster nodes, including processor utilization, memory occupancy, and I / O load, calculating the load scores of each node, and dynamically allocating small file task quotas; A file merging module for parallelly executing small file merging operations using the Spark in-memory computing engine to generate a merged file containing multiple original small files; An index recording module for creating and maintaining index information in an external database to record the location and attributes of each original small file in the merged file; A positioning access module, which is used to provide a read-write interface through a dedicated service layer to position and access the content of specific small files in a merged file.

9. A device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a method for optimizing the reading of a large number of small files as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute a method for optimizing the reading of a large number of small files as described in any one of claims 1 to 7.

Images