Neural network assistance-based file merging system and method

Abstract

The invention discloses a file merging system and method based on neural network assistance, and the system comprises an initialization component module which is used for carrying out the butt joint of historical data information, and carrying out the data sampling; the file system module is used for storing data; the business system module is used for executing a file system task; the file merging task module is used for merging small files of the file system; and the merging strategy machine module is used for monitoring the merging feature information and triggering a file merging task. According to the method, various characteristics and conditions are comprehensively considered, and the neural network and the hierarchical fuzzy analysis method are utilized to optimize the file merging strategy of the file system, so that the system performance and the resource utilization rate are greatly improved, and unnecessary resource consumption and service influence are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of file merging, and in particular is a file merging system and method based on the assistance of a neural network. Background Art

[0002] The file system optimization file merging strategy method is mainly aimed at distributed file systems, such as HDFS, because they often encounter small file problems when processing big data, which will affect storage efficiency and computing performance. However, most of the current implementations in the industry are file merging, and the implementation of file merging strategies is limited to judging whether it is executable based on resource status. These cannot effectively guarantee the stability of file merging for the entire system. We hope to propose a method that can automatically learn and trigger the small file merging strategy based on the neural network to ensure the overall stability of the system.

[0003] The following is only related to the implementation of file merging in the industry:

[0004] 1. Hive:

[0005] Hive is a data warehouse system based on Hadoop, which is used to store and process large-scale data.

[0006] In Hive, a query language similar to SQL, HiveQL (HiveSQL for short), is used to query, analyze, and report data. HiveSQL is a powerful and flexible tool that makes data operations and analysis on large data sets simple and effective. Hive can overwrite files by executing INSERTOVERWRITE, or merge ORC and RCFILE file formats by concatenate. However, it often encounters problems such as merge task failure, merge task causing business task execution time to increase, and business task execution success rate reduction.

[0007] 2. MR custom jobs:

[0008] MapReduce is a programming model and distributed computing framework that is mainly used to process large-scale data sets larger than 1TB. This model divides data into multiple small blocks and processes these small blocks in parallel on multiple computing nodes. The core idea of ​​MapReduce is divide and conquer, breaking down complex large problems into many small problems for processing. It mainly consists of two phases: Map phase and Reduce phase. In the Map phase, the input data is divided into multiple small blocks, and then these small blocks are processed in parallel to generate a set of intermediate key-value pairs. Next, in the Reduce phase, these key-value pairs are aggregated according to the key to generate the final result. Users only need to implement the two functions of map() and reduce() to achieve distributed computing.

[0009] MapReduce tasks use the CombineFileInputFormat class: CombineFileInputFormat is a built-in class of Hadoop, which can merge multiple small files into one large file. In addition, you can write a custom InputFormat class to merge small files. Specifically, you can return an array containing all the small file paths in the getSplits method of the InputFormat class, and then read these small files in the Mapper class and process them. However, executing the above tasks definitely requires resource information, and the merging of small files in the overall file system will inevitably cause the file system to be overloaded, the merge task fails, the merge task causes the business task execution time to increase, and the business task execution success rate to decrease.

[0010] None of the above solutions can achieve the effect of merging small files in each directory at the fastest speed while fully utilizing machine resources and having little impact on the normal execution of the system. Therefore, the following problems may occur:

[0011] Resource consumption: The small file merging operation itself is resource-intensive because it involves reading small files, writing the merged large file, and possible network transmission. If it is performed frequently, it may lead to excessive consumption of CPU and I / O resources.

[0012] Performance degradation: Small file merging involves a large number of disk I / O operations, which may reduce the responsiveness of the system, especially under high load or high concurrent access.

[0013] Write amplification: Small file merging may cause write amplification, that is, the amount of data actually written to the storage system is much larger than the original amount of data. This is because during the merging process, small files are rewritten into larger files.

[0014] Increased space usage: During the merge process, more storage space may be temporarily occupied because the original small files and the merged large files may exist at the same time until the merge operation is completed and the old files are cleaned up.

[0015] Impact on business operations: Since the small file merging operation takes up a lot of computing and storage resources, it may affect other running data processing jobs, causing the performance of these jobs to deteriorate.

[0016] Metadata pressure: In a distributed file system or database, frequent merging of small files increases metadata pressure because each file requires metadata to manage. Summary of the invention

[0017] In view of the above problems, the present invention is proposed to provide a neural network-assisted file merging system and method that overcomes the above problems or at least partially solves the above problems.

[0018] To achieve the above object, the present invention adopts the following technical solutions:

[0019] A neural network-assisted file merging system, the system comprising:

[0020] Initialize component module, used to connect historical data information and perform data sampling;

[0021] File system module, used to store data;

[0022] Business system module, used for executing file system tasks;

[0023] The file merging task module is used to merge small files in the file system;

[0024] The merge strategy machine module is used to monitor the merge feature information and trigger the file merge task.

[0025] Optionally, the file system module includes a Hadoop system and an FTP system.

[0026] Optionally, the business system module includes a data middle-office system.

[0027] Optionally, the merge policy machine module includes:

[0028] Monitor merge task characteristics, merge file directory characteristics, server characteristics, business task characteristics, and file system characteristics;

[0029] Use neural network ensemble training to verify the optimal execution time of merged tasks, business task execution success rate and average business task execution time;

[0030] The weight of each feature is calculated according to the hierarchical fuzzy analysis method, and finally the score is obtained according to the precipitation formula, and the sorting triggers the file merging task.

[0031] Optionally, the merge task characteristics include execution CPU configuration, execution memory configuration and execution merge task time.

[0032] Optionally, the merged file directory features include the number of valid files that need to be merged in the directory and the average size of valid files that need to be merged in the directory.

[0033] Optionally, the server characteristics include memory count, CPU core count, disk capacity, disk usage, disk IO, network IO, handle count, and handle usage.

[0034] Optionally, the business task characteristics include an average number of business task executions, a business task execution success rate, and an average business task execution time.

[0035] Optionally, the file system characteristics include the total capacity of the file system, the file system occupancy rate, and the total number of files in the current system.

[0036] The present invention also provides a neural network-assisted file merging method, which is based on any of the aforementioned neural network-assisted file merging systems, and comprises the following steps:

[0037] S1. Connect historical data execution information;

[0038] S2, regularly read various types of feature information and trigger the training of the merged strategy machine;

[0039] S3, calculating the score of each file directory;

[0040] The specific feature weight information is calculated using the hierarchical fuzzy analysis method, and the feature score is calculated based on the proportion. Finally, the following formula is defined based on the factors affecting the overall execution efficiency:

[0041]

[0042] Where: F, feature score; n, the total number of features; ω i , the weight of feature i; s i , the score of feature i; A, resource sufficiency, CPU, memory, and file system remaining resources hard rule judgment; C, customized rules, multipliers customized according to feature business requirements and strategies;

[0043] Resource adequacy reflects the current system resource usage and is used to adjust the score to adapt to the actual load of the system. Resource adequacy A is expressed as:

[0044]

[0045] Where: m, total number of resource types: CPU, memory, file system disk; f j , the remaining amount of resources; t j , total resources; p j , weight information;

[0046] S4, triggering the file merging task, sorting the file directory scores in step S3, and executing the file with the highest score first.

[0047] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0048] 1. The present invention optimizes the file merging strategy of the file system by comprehensively considering multiple characteristics and conditions and utilizing neural networks and hierarchical fuzzy analysis methods, thereby greatly improving system performance and resource utilization and reducing unnecessary resource consumption and business impact.

[0049] 2. The present invention uses neural network technology to predict and optimize small file merging strategies, taking into account characteristics in multiple dimensions, such as merging task characteristics, merging file directory characteristics, server characteristics, business task characteristics and file system characteristics, to achieve more comprehensive decision support. By training the neural network model and continuously optimizing it, it can adapt to the ever-changing system status and business needs.

[0050] 3. The present invention uses a hierarchical fuzzy analysis method to calculate feature weight information, and allows customized rules to be customized according to specific business needs and strategies through the use of customized rules, thereby increasing the flexibility and adaptability of the strategy, and adjusting the score through the concept of resource sufficiency, reflecting the current resource usage of the system, so that the merged strategy can adapt to different system loads.

[0051] 4. The present invention utilizes neural network technology to automatically learn and predict project execution status, and realize intelligent scheduling and merging tasks without human intervention; it takes into account characteristics of multiple dimensions, such as merged task characteristics, merged file directory characteristics, server characteristics, business task characteristics and file system characteristics, to achieve more comprehensive decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a neural network-assisted file merging system process provided in an embodiment of the present application;

[0053] Figure 2 A flowchart of a neural network-assisted file merging method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] Embodiment 1:

[0056] See also Figure 1 This embodiment provides a file merging system based on neural network assistance, the system comprising:

[0057] Initialize the component module to connect to historical data information and perform data sampling.

[0058] The initialization component module is used to connect to historical data information and perform data sampling based on the content in the current file system to form a file synchronization test script for generating training data for the merge strategy machine.

[0059] File system module, used to store data.

[0060] The file system module includes the Hadoop system and the FTP system.

[0061] Business system module, used to execute file system tasks.

[0062] The business system module includes the data middle platform system.

[0063] The file merging task module is used to merge small files in the file system. This task has no restrictions and can be a custom MR, Hive SQL, and other tasks.

[0064] The merge strategy machine module is used to monitor the merge feature information and trigger the file merge task.

[0065] The merge strategy machine module includes:

[0066] Monitors merge task characteristics, merge file directory characteristics, server characteristics, business task characteristics, and file system characteristics.

[0067] Use neural network ensemble training to verify the optimal execution time of merged tasks, business task execution success rate and average business task execution time.

[0068] The weight of each feature is calculated according to the hierarchical fuzzy analysis method, and finally the score is obtained according to the precipitation formula, and the sorting triggers the file merging task.

[0069] The merge task characteristics include execution CPU configuration, execution memory configuration, and execution merge task time.

[0070] The merged file directory characteristics include the number of valid files that need to be merged in the directory and the average size of valid files that need to be merged in the directory.

[0071] Server characteristics include memory count, CPU core count, disk capacity, disk usage, disk IO, network IO, handle count, and handle usage.

[0072] Business task characteristics include the average number of business task executions, the business task execution success rate, and the average business task execution time.

[0073] File system characteristics include the total file system capacity, file system occupancy, and the total number of files in the current system.

[0074] This embodiment comprehensively considers various characteristics and conditions, and uses neural network and hierarchical fuzzy analysis to optimize the file merging strategy of the file system, so as to improve system performance and resource utilization, and reduce unnecessary resource consumption and business impact.

[0075] This embodiment uses neural network technology to predict and optimize small file merging strategies, taking into account features in multiple dimensions, such as merging task features, merged file directory features, server features, business task features, and file system features, to achieve more comprehensive decision support. By training the neural network model and continuously optimizing it, it can adapt to the ever-changing system status and business needs.

[0076] This embodiment uses a hierarchical fuzzy analysis method to calculate feature weight information, and allows customized rules to be customized according to specific business needs and strategies through the use of customized rules, thereby increasing the flexibility and adaptability of the strategy, and adjusting the score through the concept of resource sufficiency, reflecting the current resource usage of the system, so that the merged strategy can adapt to different system loads.

[0077] This embodiment uses neural network technology to automatically learn and predict project execution status, and realize intelligent scheduling of merging tasks without human intervention; it takes into account characteristics of multiple dimensions, such as merging task characteristics, merging file directory characteristics, server characteristics, business task characteristics and file system characteristics, to achieve more comprehensive decision support.

[0078] Embodiment 2:

[0079] Embodiment 2 discloses a neural network-assisted file merging method, which is based on the above-mentioned neural network-assisted file merging system, and includes the following steps:

[0080] S1. Connect historical data execution information.

[0081] When connecting to an existing file system, it is necessary to connect to its historical data execution information for merging the policy machine training data. If there is no historical data or the record is too small, we will configure the test script in batches according to the on-site server environment, on-site business system, and file system to execute task execution information under different environmental states and collect execution information.

[0082] S2. Read various types of feature information regularly and trigger the training of the merged strategy machine.

[0083] Acquire the merge task characteristics, merge file directory characteristics, server characteristics, task characteristics, and file system characteristics according to the configured time timing.

[0084] After the initialization component has executed all the task information that needs to be collected, the execution record information is stored for the merged strategy machine training. Here, a set of neural networks [including multi-layer perceptrons and recurrent neural networks] are used for simultaneous training. In the subsequent continuous optimization process, whoever trains better will give the predicted regression data. There are three types of predicted data here: average business task execution time [regression task], business task execution success rate [regression task], and execution merged task time [regression task].

[0085] S3. Calculate the score of each file directory.

[0086] Here, the score is calculated by a customized algorithm. Based on the experience of all projects since the establishment of the company and the expert scores, we obtain the main features that ensure the stable operation of the system when the files are merged [all the features mentioned above]. And use the hierarchical fuzzy analysis method to calculate the specific feature weight information, and calculate the feature score according to the proportion [if it is indivisible data (such as file merging execution time), divide it into 20 dimensions in equal proportion according to the recorded information during the statistical period, and score it in percentage]. Finally, define the following formula based on the factors affecting the overall execution efficiency:

[0087]

[0088] Where: F, feature score; n, the total number of features; ω i , the weight of feature i; s i , the score of feature i; A, resource sufficiency, CPU, memory, and file system remaining resources hard rule judgment; C, customized rules, multipliers customized according to feature business requirements and strategies;

[0089] Resource adequacy reflects the current system resource usage and is used to adjust the score to adapt to the actual load of the system. Resource adequacy A is expressed as:

[0090]

[0091] Where: m, total number of resource types: CPU, memory, file system disk; f j , the remaining amount of resources; t j , total resources; p j , weight information, normalization results of three feature weights.

[0092] S4, trigger the file merging task, sort the file directory scores in step S3, and execute the file with higher scores first. When the score is 0, terminate the submission task and wait for the configured time. If the recalculated score is greater than 0, continue to execute according to the sorting.

[0093] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A neural network-assisted file merging system, characterized in that: The system comprises: Initialize the component module to connect to historical data information and perform data sampling; File system module, used to store data; Business system module, used for executing file system tasks; The file merging task module is used to merge small files in the file system; The merge strategy machine module is used to monitor the merge feature information and trigger the file merge task.

2. A neural network-assisted file merging system as claimed in claim 1, characterized in that: The file system module includes Hadoop system and FTP system.

3. A neural network-assisted file merging system as claimed in claim 1, characterized in that: The business system module includes the data middle platform system.

4. A neural network-assisted file merging system as claimed in claim 1, characterized in that: The merge strategy machine module includes: Monitor merge task characteristics, merge file directory characteristics, server characteristics, business task characteristics, and file system characteristics; Use neural network ensemble training to verify the optimal execution time of merged tasks, business task execution success rate and average business task execution time; The weight of each feature is calculated according to the hierarchical fuzzy analysis method, and finally the score is obtained according to the precipitation formula, and the sorting triggers the file merging task.

5. A neural network-assisted file merging system as claimed in claim 4, characterized in that: The merge task characteristics include execution CPU configuration, execution memory configuration and execution merge task time.

6. A neural network-assisted file merging system as claimed in claim 4, characterized in that: The merged file directory features include the number of valid files that need to be merged in the directory and the average size of the valid files that need to be merged in the directory.

7. A neural network-assisted file merging system as claimed in claim 4, characterized in that: The server characteristics include memory number, CPU core number, disk capacity, disk usage, disk IO, network IO, handle number and handle usage.

8. The neural network-assisted file merging system according to claim 4, characterized in that: The business task characteristics include the average number of business task executions, the business task execution success rate, and the average business task execution time.

9. The neural network-assisted file merging system according to claim 4, characterized in that: The file system characteristics include the total capacity of the file system, the file system occupancy rate and the total number of files in the current system.

10. A method for merging files assisted by a neural network, the method being based on a system for merging files assisted by a neural network as claimed in any one of claims 1 to 9, characterized in that: The method comprises the following steps: S1. Connect historical data execution information; S2, regularly read various types of feature information and trigger the training of the merged strategy machine; S3, calculating the score of each file directory; The specific feature weight information is calculated using the hierarchical fuzzy analysis method, and the feature score is calculated based on the proportion. Finally, the following formula is defined based on the factors affecting the overall execution efficiency: Where: F, feature score; n, the total number of features; ω i , the weight of feature i; s i , the score of feature i; A, resource sufficiency, CPU, memory, and file system remaining resources hard rule judgment; C, customized rules, multipliers customized according to feature business requirements and strategies; Resource adequacy reflects the current system resource usage and is used to adjust the score to adapt to the actual load of the system. Resource adequacy A is expressed as: Where: m, total number of resource types: CPU, memory, file system disk; f j , the remaining amount of resources; t j , total resources; p j , weight information; S4, triggering the file merging task, sorting the file directory scores in step S3, and executing the file with the highest score first.

Images