A parallel file parsing method based on MCF system

Abstract

The present invention discloses a parallel file parsing processing method and system based on an MCF system, and relates to the technical field of parallel file parsing processing, including receiving file data, slicing the file; generating data checksums and removing data redundancy, parsing and processing according to the file format; storing data, and optimizing the controller load. The parallel file parsing processing method based on the MCF system provided by the present invention can intelligently manage the parsing process of different types of files through dynamic adjustment of file type, size and controller load, thereby improving the parsing efficiency. By dynamically adjusting the number of shards and distributed storage, the efficient operation of the system is ensured under high load conditions, single point overload is avoided, and the allocation of storage resources is optimized. The present invention achieves better results in terms of processing efficiency, scope of application, and dynamic adjustment.

Description

Technical Field

[0001] The present invention relates to the technical field of parallel file parsing and processing, and in particular to a parallel file parsing and processing method and system based on an MCF system. Background Art

[0002] With the continuous advancement of information technology, distributed computing and storage technology has been widely used in the field of large-scale data processing. As an efficient file storage and processing method, parallel file system realizes efficient data storage and access through the collaborative work of multiple computers. In recent years, the rapid development of cloud computing, virtualization technology and distributed computing platforms has made parallel file systems more widely used, especially in the fields of processing massive data, video data, and large file storage. Traditional parallel file processing methods have improved data storage efficiency to a certain extent, but they still face challenges such as uneven load, slow data recovery, and storage redundancy. Therefore, how to improve file parsing efficiency and storage management capabilities while ensuring data security, especially in high-load and massive data environments, has become a hot topic in current research.

[0003] Existing parallel file systems have multiple limitations in file parsing and storage management, especially in high-load environments, which greatly affect the performance and efficiency of the system. First of all, traditional file sharding processing methods are mostly based on fixed shard size and total file size for calculation, and lack adaptability to dynamic changes in system load. This means that when the load is too high, the system often cannot intelligently adjust the number of shards, resulting in waste of resources or controller overload, and cannot provide efficient file processing capabilities under different load scenarios. Most existing technologies use static sharding and load balancing strategies. This method fails to effectively combine the real-time load and task requirements of the controller, and easily causes performance bottlenecks in the system.

[0004] Secondly, traditional verification and data redundancy processing technologies mainly rely on simple linear verification code generation methods, which poses challenges to large-scale data recovery in high-load systems. Although existing technologies use erasure coding technology to enhance redundancy, in a distributed environment, the verification code generation method is too fixed and difficult to dynamically adjust based on factors such as real-time data quality and controller load, resulting in unbalanced redundant storage and low data recovery efficiency. Especially in the case of loss of certain controllers or shards, the data recovery process still takes a long time, affecting the reliability and fault tolerance of the overall system.

[0005] In addition, existing file parsing methods mostly rely on static parsing strategies, which lack flexibility and adaptability when facing files of different types and sizes. For example, when processing files of different formats such as text, images, and videos, the parsing time and processing methods are often fixed, and the impact of file type, file size, and system load on parsing efficiency is not considered. This greatly limits the speed of file parsing and system response time in a high-load environment, resulting in the inability to effectively improve the overall performance of the system. Summary of the invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is: the existing parallel file processing methods have problems such as uneven load, low data recovery efficiency, static file parsing process, uneven redundant storage, and how to intelligently adjust the number of shards, optimize the generation of verification shards, and improve file parsing efficiency under high load environments.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: a parallel file parsing and processing method based on the MCF system, comprising receiving file data and slicing the file; generating data checksums and removing data redundancy, parsing and processing according to the file format; storing data and optimizing the controller load.

[0009] As a preferred solution of the parallel file parsing method based on the MCF system described in the present invention, wherein: the receiving of file data includes the user sending the file to the system through the client, and the file is received by the main control node.

[0010] The main control node checks the format, file size and integrity of the file, and identifies the file as meeting the processing requirements if it meets the preset requirements, and confirms whether the file meets the processing requirements.

[0011] Extract file metadata, including file type and file size .

[0012] As a preferred solution of the parallel file parsing method based on the MCF system described in the present invention, the file segmentation includes: and the default shard size , determine the current number of shards through dynamic adjustment model , expressed as:

[0013] ;

[0014] in, Indicates the load of the controller, Indicates the maximum processing capability of the controller.

[0015] As a preferred solution of the parallel file parsing method based on the MCF system described in the present invention, wherein: the generating of data checksum and removing of data redundancy includes each data fragment A corresponding checksum fragment will be generated based on the data size, storage location and quality The check shards are characterized by the characteristics of the data shards and the current load of the controller. The generated check shards are dynamically weighted according to the characteristics of each data shard through the Sigmoid activation function. After the check shards are generated, they are stored on different controllers. By distributing the check shards on multiple controllers, when a controller or data shard is lost, the remaining check shards can be used to restore the lost data.

[0016] As a preferred solution of the parallel file parsing method based on the MCF system described in the present invention, the parsing according to the file format includes identifying the file as a text file, an image file, and a video file according to the file format, and performing the parsing according to the file size. and the controller load , adjust the parsing time during the parsing process, expressed as:

[0017] ;

[0018] Among them, ParsingTime For File The analysis time, is the fragment size, As the basic analytical time coefficient, To control the file size impact index, is the load regulation factor, is the file type influence coefficient, is the file type weight.

[0019] As a preferred solution of the parallel file parsing method based on the MCF system described in the present invention, the data storage includes slicing each file, and the system records the file location, type, size, and slicing hash value.

[0020] A hash value is calculated for each shard and will be stored in the metadata.

[0021] As a preferred solution of the parallel file parsing processing method based on the MCF system of the present invention, wherein: the optimization of the controller load includes calculating the load weight according to the number of tasks and processing capacity of each controller , expressed as:

[0022] ;

[0023] in, For controller The load weight, For controller The current number of tasks, For controller processing capacity, k is the load optimization adjustment coefficient, is the Sigmoid activation function.

[0024] Another object of the present invention is to provide a parallel file parsing and processing system based on the MCF system, which can solve the problem of balancing redundant storage by dynamically generating and distributing the verification fragments, so that the data has higher reliability and recovery ability during the storage process, and solves the problem of uneven redundant storage in the current parallel file processing method.

[0025] As a preferred solution of the parallel file parsing and processing system based on the MCF system described in the present invention, it includes a data processing module, a data parsing module, and a data storage module. The data processing module is used to receive file data and slice the file. The data parsing module is used to generate data check codes and remove data redundancy, and perform parsing and processing according to the file format. The data storage module is used to store data and optimize the controller load.

[0026] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a parallel file parsing processing method based on an MCF system.

[0027] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a parallel file parsing processing method based on an MCF system.

[0028] Beneficial effects of the present invention: The parallel file parsing processing method based on the MCF system provided by the present invention can intelligently manage the parsing process of different types of files through dynamic adjustment of file type, size and controller load, thereby improving parsing efficiency. By dynamically adjusting the number of shards and distributed storage, efficient operation of the system under high load conditions is ensured, single point overload is avoided and the allocation of storage resources is optimized. Through efficient metadata management, file shards can be quickly retrieved, especially when performing file recovery or migration. The accurate recording of metadata makes it more efficient to locate file shards, effectively improving the speed of file storage and management, and ensuring storage efficiency and access efficiency in large-scale data processing scenarios. The present invention achieves better results in terms of processing efficiency, scope of application and dynamic adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1 This is an overall flow chart of a parallel file parsing method based on an MCF system provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0032] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a parallel file parsing processing method based on the MCF system, comprising:

[0033] S1: Receive file data and split the file into pieces.

[0034] Furthermore, receiving the file data includes a user sending the file to the system through a client, and the file is received by the main control node.

[0035] The main control node checks the format, file size and integrity of the file, and identifies the file as meeting the processing requirements if it meets the preset requirements, and confirms whether the file meets the processing requirements.

[0036] Extract file metadata, including file type and file size .

[0037] It should be noted that file sharding is included in the file storage system. The main goal of file sharding and allocation is to cut a large file into multiple smaller shards according to the size of the file and the current controller load, and dynamically adjust the number of allocated shards according to the real-time load of the controller. Through this operation, the system can store and process files in parallel on multiple controllers to ensure load balancing.

[0038] The number of shards is dynamically adjusted based on the controller load. When the controller load is high, the system will increase the number of shards to share a greater processing pressure.

[0039] When the controller load is low, the number of shards will be reduced to avoid unnecessary waste of resources.

[0040] The number of shards is dynamically calculated based on the file size and the real-time load of the controller, allowing the system to flexibly allocate tasks based on the load and optimize resource usage. and the default shard size , determine the current number of shards through a dynamic adjustment model , expressed as:

[0041] ;

[0042] in, Indicates the load of the controller, Indicates the maximum processing capability of the controller.

[0043] S2: Generate data checksum and remove data redundancy, and perform parsing according to the file format.

[0044] Furthermore, data checksums are generated and data redundancy is removed including each data fragment. A corresponding checksum fragment will be generated based on the data size, storage location and quality The check shards are characterized by the characteristics of the data shards and the current load of the controller. The generated check shards are dynamically weighted according to the characteristics of each data shard through the Sigmoid activation function. This nonlinear adjustment can ensure that more redundant check data is generated when the system load is high (that is, when the controller load is heavy), and reduce redundant check when the system load is low, optimizing the use of storage and computing resources. After the check shards are generated, they are stored on different controllers. By distributing the check shards on multiple controllers, when a controller or data shard is lost, the remaining check shards are used to restore the lost data. The check shards not only play the role of redundant backup when storing data, but also provide necessary support when reading and restoring data. According to the read request, the system obtains the location of the check shards through the metadata management module, and uses these check shards to ensure data integrity during data recovery or reconstruction. The storage management strategy of the check shards is consistent with the data shards, and is also dynamically adjusted through the load balancing and task scheduling mechanism to ensure the reasonable distribution of storage tasks among different controllers.

[0045] It should be noted that parsing the file format includes identifying the file format as a text file, an image file, or a video file, and performing the parsing according to the file size. and the controller load , adjust the parsing time during the parsing process, expressed as:

[0046] ;

[0047] Among them, ParsingTime For File The analysis time, is the fragment size, As the basic analytical time coefficient, To control the file size impact index, is the load regulation factor, is the file type influence coefficient, is the file type weight.

[0048] Text file parsing: For text files, the system converts the file content into a processable data structure through steps such as word segmentation, syntax analysis, and index generation. The file content is scanned line by line or byte by byte, and information is extracted according to grammatical rules.

[0049] Image file parsing: For image files, the parsing process involves steps such as image compression, feature extraction, and edge detection. Through image processing algorithms, the key features of the image are extracted, which can be further used in applications such as image compression and image content retrieval.

[0050] Video file parsing: For video files, the parsing process usually includes video decoding, frame extraction, key frame extraction, etc. The system will parse the video data frame by frame and extract key frames or perform compression conversion for storage or further processing.

[0051] S3: Store data and optimize controller load.

[0052] Furthermore, data storage includes sharding each file, and the system records the file location, type, size, and shard hash value.

[0053] A hash value is calculated for each shard and will be stored in the metadata.

[0054] Specifically, during the file sharding process, the system will record detailed metadata for each shard. These metadata contain basic information about the file shard, such as file location, type, size, checksum information, etc. Each file shard will be mapped to a storage location in the system and the corresponding attributes will be recorded. By recording this information about the file shard, the system can quickly locate a shard and perform operations when needed.

[0055] To ensure the integrity and consistency of files, the system calculates a hash value (such as SHA-256) for each file shard and stores the hash value in the metadata. The hash value is the unique identifier of the file shard, which ensures that data will not be tampered with or lost during file storage, transmission, and recovery. When a file or shard is damaged, the system can quickly identify and trigger a repair mechanism by comparing the hash value, thereby improving the efficiency of data recovery.

[0056] In order to improve query efficiency, the system uses a multi-level index structure to organize and store metadata. Multi-level indexing can quickly locate the location and related information of file shards in a large-scale distributed environment. On the basis of indexing, the system also uses a weighted storage mechanism to give priority to the storage of certain frequently accessed or important file shards to improve their access speed. This storage mechanism can avoid the performance bottleneck caused by accessing hot data, thereby optimizing the entire file storage and retrieval process. In order to speed up data retrieval, the system uses a multi-level index structure to manage metadata. Multi-level indexing divides metadata into different levels, and each level is responsible for indexing different attributes or file characteristics. Through the weighted storage mechanism, the system will give priority to storing some frequently accessed or important shards in high-performance storage devices to reduce access latency and improve the response speed of the system. For example, hot data may be stored on SSD devices for fast reading.

[0057] It should be noted that the goal of controller load and task scheduling is to dynamically adjust the distribution of tasks according to the current load of the system to ensure that the load between controllers remains balanced. Through accurate load balancing algorithms, the system can avoid excessive concentration of tasks on a certain controller, thereby avoiding performance bottlenecks caused by overload. Optimizing controller load includes calculating load weights based on the number of tasks and processing power of each controller. , expressed as:

[0058] ;

[0059] in, For controller The load weight, For controller The current number of tasks, For controller processing capacity, k is the load optimization adjustment coefficient, is the Sigmoid activation function.

[0060] Embodiment 2, an embodiment of the present invention, provides a parallel file parsing method based on the MCF system. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0061] First, in order to verify the effectiveness of the parallel file parsing method based on the MCF system, the following experiment was conducted. This experiment was based on a data set containing 100GB files, simulating the large-scale data storage and parsing process, and experimenting with the steps of file reception, sharding, verification, parsing, storage and load optimization.

[0062] Hardware configuration: The experiment uses 5 high-performance computing nodes, each with an 8-core CPU, 64GB memory, 1TB hard disk capacity, and 1Gbps network bandwidth. These nodes form an MCF system, in which each node acts as a controller to process and store shard data.

[0063] Data preparation: Three types of files were selected for the experiment: text files (10GB), image files (30GB), and video files (60GB). The processing of each file type was optimized through a specific parsing method.

[0064] The shard size is set to 2GB.

[0065] Table 1 Experimental data comparison table

[0066]

[0067] In Experiment 1 (text file processing) and Experiment 4, the system maintained a low load weight (1.2 and 1.1 respectively) by dynamically adjusting the load of the controller, and the parsing time and data storage time were reasonably controlled. Compared with the traditional load distribution method, the present invention shows better stability and efficiency in load optimization.

[0068] In Experiment 3 and Experiment 6, the video files were large, and the system dynamically adjusted the number of shards (30 shards) and the number of check shards (10 shards) according to the file size and controller load. By introducing a dynamic adjustment model, the adjustment of the number of shards ensures load balancing between controllers and avoids overloading of some controllers. Compared with the traditional method of statically fixing the number of shards, the present invention can be flexibly adjusted according to the actual load conditions, thereby improving the efficiency of resource utilization.

[0069] In Experiment 2 and Experiment 5, although the parsing time of image files and text files (200 seconds and 190 seconds respectively) was relatively long, the parsing process was effectively associated with the load and file type through dynamic adjustment of the system, avoiding excessive waste of computing resources. The adjustment of parsing time depends on the file type and controller load, ensuring efficient processing of files and efficient use of system resources.

[0070] The generation and redundant storage of check shards are crucial for file recovery. Experimental data shows that the combination of the number of check shards generated (up to 10) and the controller load and data shard quality effectively improves the flexibility of redundant storage. Under high load conditions, the system can use the remaining check shards to quickly recover lost data, significantly enhancing the system's fault tolerance.

[0071] Embodiment 3, an embodiment of the present invention provides a parallel file parsing and processing system based on the MCF system, including a data processing module, a data parsing module, and a data storage module.

[0072] The data processing module is used to receive file data and segment the file. The data analysis module is used to generate data verification codes and remove data redundancy, and perform analysis and processing according to the file format. The data storage module is used to store data and optimize the controller load.

[0073] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0074] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0075] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk case (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0076] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logical function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0077] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A parallel file parsing method based on MCF system, characterized in that: include: Receive file data and split the file into pieces; Generate data checksums and remove data redundancy, and perform parsing according to the file format; Perform data storage and optimize controller load; The file segmentation includes segmenting the file according to the total size of the file. and the default shard size , determine the current number of shards through dynamic adjustment model , expressed as: ； in, Indicates the load of the controller, Indicates the maximum processing capacity of the controller. Indicates the fragment size control coefficient; The parsing process according to the file format includes identifying the file format as a text file, an image file, and a video file, and performing the parsing process according to the file size. and the controller load , adjust the parsing time during the parsing process, expressed as: ； Among them, ParsingTime For File The analysis time, is the fragment size, As the basic analytical time coefficient, To control the file size impact index, is the load regulation factor, is the file type influence coefficient, is the file type weight; The optimizing controller load includes calculating the load weight according to the number of tasks and processing capacity of each controller. , expressed as: ； in, For controller The load weight, For controller The current number of tasks, For controller processing capacity, k is the load optimization adjustment coefficient, is the Sigmoid activation function.

2. The parallel file parsing and processing method based on the MCF system as claimed in claim 1, characterized in that: The receiving of file data includes a user sending a file to the system through a client, and the file is received by the main control node; The main control node checks the format, file size and integrity of the file, and identifies the file that meets the preset requirements as meeting the processing requirements to confirm whether the file meets the processing requirements; Extract file metadata, including file type and file size .

3. The parallel file parsing and processing method based on the MCF system as claimed in claim 2, characterized in that: The generating of data checksum and removing of data redundancy includes each data fragment A corresponding checksum fragment will be generated based on the data size, storage location and quality The check shards are characterized by the characteristics of the data shards and the current load of the controller. The generated check shards are dynamically weighted according to the characteristics of each data shard through the Sigmoid activation function. After the check shards are generated, they are stored on different controllers. By distributing the check shards on multiple controllers, when a controller or data shard is lost, the remaining check shards can be used to restore the lost data.

4. The parallel file parsing method based on the MCF system as claimed in claim 3, characterized in that: The data storage includes slicing each file, and the system records the file location, type, size, and slicing hash value; A hash value is calculated for each shard and will be stored in the metadata.

5. A system using the parallel file parsing method based on the MCF system as claimed in any one of claims 1 to 4, characterized in that: It includes data processing module, data parsing module and data storage module; The data processing module is used to receive file data and slice the file; The data parsing module is used to generate data check codes and remove data redundancy, and perform parsing processing according to the file format; The data storage module is used for data storage and optimizing controller load.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the parallel file parsing processing method based on the MCF system according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the parallel file parsing processing method based on the MCF system according to any one of claims 1 to 4 are implemented.

Images