A method and system for processing electronic file data based on big data

Abstract

The present invention proposes an electronic file data processing method and system based on big data, which relates to the field of data processing technology, obtains electronic file data, pre-processes and splits the electronic file data into categories, obtains file split data packets, and then obtains distributed storage data packets; performs data processing capability analysis on the obtained preset service cluster, performs processed capability analysis on the distributed storage data packets, performs simulated processing service analysis respectively, and then performs simulated processing judgment on the distributed storage data packets; obtains cluster processing ranking and data packet ranking, performs actual processing and update processing on the distributed storage data packets through preset service clusters with the same ranking, and obtains processing results. The present invention realizes comprehensive management and optimized processing of electronic file data.

Description

Technical Field

[0001] The present invention proposes a method and system for processing electronic file data based on big data, and relates to the field of data processing technology, and specifically to the field of electronic file data processing technology based on big data. Background Art

[0002] The processing of electronic documents involves multiple links such as data collection, storage, management, analysis and utilization, and the demand for processing massive electronic document data is increasing. However, traditional electronic document processing methods face many challenges in terms of processing efficiency, resource utilization and scalability. Traditional electronic document processing methods usually adopt a centralized storage and processing mode, that is, electronic document data is stored in a single server or data center and processed through the server or data center. However, with the explosive growth of electronic document data, the centralized storage and processing mode has gradually exposed problems such as limited storage capacity, slow processing speed, low resource utilization, and poor scalability. Existing technologies make it difficult to analyze the processing capabilities of service clusters and data packets individually and in combination, resulting in low efficiency in electronic document processing and waste of processing resources. Summary of the invention

[0003] The present invention provides a method and system for processing electronic file data based on big data to solve the above problems:

[0004] The present invention proposes a method and system for processing electronic file data based on big data, the processing method comprising:

[0005] S1. Acquire electronic file data, pre-process and split the electronic file data into categories, obtain file split data packets, and then obtain distributed storage data packets;

[0006] S2, analyzing the data processing capability of the acquired preset service cluster, allocating distributed storage data packets according to the evaluation results, performing simulated processing service analysis on the preset allocated data packets, and then performing simulated processing judgment on the preset service cluster;

[0007] S3, analyzing the processing capacity of the distributed storage data packets, assigning preset service clusters to the distributed storage data packets, performing simulation processing service analysis, and then performing simulation processing determination on the distributed storage data packets;

[0008] S4. Obtain cluster processing ranking and data packet ranking, and perform actual processing and update processing on the distributed storage data packet through the preset service cluster with the same ranking to obtain the processing result.

[0009] Furthermore, the S1 includes:

[0010] Acquire electronic file processing request information, and acquire electronic file data that needs to be processed according to the electronic file processing request information;

[0011] Preprocessing the electronic file data to obtain preprocessed electronic file data;

[0012] Acquire preset electronic file splitting category information, and perform file data splitting on the preprocessed electronic file data according to the preset electronic file splitting category information to obtain a file splitting data packet;

[0013] Each file is split into data packets for distributed storage to obtain multiple distributed storage data packets.

[0014] Further, the S2 includes:

[0015] Acquire information of multiple preset service clusters, and calculate a data processing capacity coefficient of each preset service cluster according to the preset service cluster information;

[0016] Allocating distributed storage data packets to the preset service cluster according to the data processing capacity coefficient;

[0017] Obtaining a preset allocation data packet of a preset service cluster;

[0018] Performing simulation processing services on the preset allocated data packets through a preset service cluster to obtain cluster simulation processing data;

[0019] Calculating a cluster simulation processing coefficient according to the cluster simulation processing data;

[0020] A simulation processing determination is performed on the preset service cluster according to the cluster simulation processing coefficient to obtain simulation processing determination information of the preset service cluster.

[0021] Further, the S3 includes:

[0022] Distributed storage data packet information, calculating a data packet processing capacity coefficient of each distributed storage data packet according to the distributed storage data packet information;

[0023] Allocating a preset service cluster to the distributed storage data packet according to the data packet processing capacity coefficient;

[0024] Obtain a data packet cluster for distributed storage of data packets;

[0025] Performing simulation processing services on the distributed storage data packets through the data packet cluster to obtain data packet simulation processing data;

[0026] Calculating a data packet simulation processing coefficient according to the data packet simulation processing data;

[0027] The distributed storage data packet is subjected to simulation processing determination according to the data packet simulation processing coefficient to obtain simulation processing determination information of the distributed storage data packet.

[0028] Further, the S4 includes:

[0029] Sort the preset service clusters from large to small by cluster simulation processing coefficients to obtain cluster processing ranking;

[0030] The distributed storage data packets are sorted from large to small by using the data packet simulation processing coefficient to obtain the data packet sorting;

[0031] The distributed storage data packets are actually processed through preset service clusters of equal ranking to obtain actual processing data;

[0032] Compare the actual processing data with the preset processing threshold value to obtain a processing comparison result;

[0033] The cluster processing order and the data packet order are updated according to the processing comparison result, and then actual reprocessing is performed to obtain a reprocessing result.

[0034] Furthermore, the processing system comprises:

[0035] A file data splitting module is used to obtain electronic file data, pre-process and classify the electronic file data, obtain file split data packets, and then obtain distributed storage data packets;

[0036] The cluster capability analysis module is used to analyze the data processing capability of the acquired preset service cluster, allocate distributed storage data packets according to the evaluation results, perform simulated processing service analysis on the preset allocated data packets, and then perform simulated processing judgment on the preset service cluster;

[0037] The data packet capability analysis module is used to analyze the processing capability of distributed storage data packets, allocate preset service clusters to distributed storage data packets, perform simulated processing service analysis, and then perform simulated processing judgment on distributed storage data packets;

[0038] The actual processing module is used to obtain cluster processing ranking and data packet ranking, and to perform actual processing and update processing on distributed storage data packets through preset service clusters with the same ranking to obtain processing results.

[0039] Furthermore, the file data splitting module includes:

[0040] Acquire electronic file processing request information, and acquire electronic file data that needs to be processed according to the electronic file processing request information;

[0041] Preprocessing the electronic file data to obtain preprocessed electronic file data;

[0042] Acquire preset electronic file splitting category information, and perform file data splitting on the preprocessed electronic file data according to the preset electronic file splitting category information to obtain a file splitting data packet;

[0043] Each file is split into data packets for distributed storage to obtain multiple distributed storage data packets.

[0044] Furthermore, the cluster capability analysis module includes:

[0045] Acquire information of multiple preset service clusters, and calculate a data processing capacity coefficient of each preset service cluster according to the preset service cluster information;

[0046] Allocating distributed storage data packets to the preset service cluster according to the data processing capacity coefficient;

[0047] Obtaining a preset allocation data packet of a preset service cluster;

[0048] Performing simulation processing services on the preset allocated data packets through a preset service cluster to obtain cluster simulation processing data;

[0049] Calculating a cluster simulation processing coefficient according to the cluster simulation processing data;

[0050] A simulation processing determination is performed on the preset service cluster according to the cluster simulation processing coefficient to obtain simulation processing determination information of the preset service cluster.

[0051] Furthermore, the data packet capability analysis module includes:

[0052] Distributed storage data packet information, calculating a data packet processing capacity coefficient of each distributed storage data packet according to the distributed storage data packet information;

[0053] Allocating a preset service cluster to the distributed storage data packet according to the data packet processing capacity coefficient;

[0054] Obtain a data packet cluster for distributed storage of data packets;

[0055] Performing simulation processing services on the distributed storage data packets through the data packet cluster to obtain data packet simulation processing data;

[0056] Calculating a data packet simulation processing coefficient according to the data packet simulation processing data;

[0057] The distributed storage data packet is subjected to simulation processing determination according to the data packet simulation processing coefficient to obtain simulation processing determination information of the distributed storage data packet.

[0058] Furthermore, the actual processing module includes:

[0059] Sort the preset service clusters from large to small by cluster simulation processing coefficients to obtain cluster processing ranking;

[0060] The distributed storage data packets are sorted from large to small by using the data packet simulation processing coefficient to obtain the data packet sorting;

[0061] The distributed storage data packets are actually processed through preset service clusters of equal ranking to obtain actual processing data;

[0062] Compare the actual processing data with the preset processing threshold value to obtain a processing comparison result;

[0063] The cluster processing order and the data packet order are updated according to the processing comparison result, and then actual reprocessing is performed to obtain a reprocessing result.

[0064] Beneficial effects of the present invention: Through preprocessing and category splitting, redundant data in the processing process is reduced and processing efficiency is improved. Distributed storage and parallel processing enable multiple service cluster nodes to process different data packets at the same time, further improving the processing speed. According to the performance evaluation results of the service cluster and the data packet, resources are reasonably allocated to avoid resource waste and bottleneck problems. Simulation processing service analysis helps to discover potential problems before actual processing, so as to optimize and adjust in advance. Data cleaning and format conversion in the preprocessing step ensure the accuracy and consistency of the input data, and can improve the accuracy of the processing results. Multiple iterative processing and update processing can continuously correct and improve the processing results to make them more in line with actual needs. The distributed storage and parallel processing architecture allows the system to easily expand new service cluster nodes and data storage nodes to meet the growing processing needs. The system can flexibly adapt to different processing tasks and data types, improving the versatility and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The invention is intended to be a method for processing electronic document data based on big data;

[0066] Figure 2 A schematic diagram for allocating data packets for distributed storage;

[0067] Figure 3 Assign a diagram to the preset service cluster. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0069] In one embodiment of the present invention, a method and system for processing electronic file data based on big data is proposed by the present invention, and the processing method includes:

[0070] S1. Acquire electronic file data, pre-process and split the electronic file data into categories, obtain file split data packets, and then obtain distributed storage data packets;

[0071] S2, analyzing the data processing capability of the acquired preset service cluster, allocating distributed storage data packets according to the evaluation results, performing simulated processing service analysis on the preset allocated data packets, and then performing simulated processing judgment on the preset service cluster;

[0072] S3, analyzing the processing capacity of the distributed storage data packets, assigning preset service clusters to the distributed storage data packets, performing simulation processing service analysis, and then performing simulation processing determination on the distributed storage data packets;

[0073] S4, obtain cluster processing ranking and data packet ranking, and perform actual processing and update processing on the distributed storage data packet through the preset service cluster with the same ranking to obtain the processing result, such as Figure 1 shown.

[0074] The working principle of the above technical solution is as follows: electronic file data is obtained from various sources (such as databases, file servers, networks, etc.). The electronic file data is preprocessed, including data cleaning (removing irrelevant information, correcting erroneous data, etc.), format conversion (unifying file formats), category splitting (dividing files into different categories according to their content or attributes), etc., to obtain file split data packets. The file split data packets are further processed into distributed storage data packets for storage and access on multiple storage nodes. The performance of data packet processing is evaluated for a preset service cluster (composed of multiple computing nodes). According to the evaluation results, data packets are allocated to service cluster nodes. Simulated processing service analysis is performed on the preset allocated data packets, and the preliminary processing capacity of the service cluster is evaluated by simulating the running of processing tasks, and a simulated processing judgment is made. The processing capacity of the distributed storage data packets is analyzed, that is, the impact of factors such as the size and complexity of the data packets on the processing capacity is evaluated. According to the analysis results, the preset service cluster nodes are allocated to the data packets, and the simulated data processing capacity of the data packets is evaluated. Simulated processing service analysis is performed on the allocated data packets, and a simulated processing judgment is made. Get the cluster processing ranking and data packet ranking, and select the preset service cluster with the same ranking to actually process the distributed storage data packet according to the ranking results. During the processing, the data packet needs to be iterated or updated multiple times to obtain the final processing result. The processing result may include data analysis reports, processed data files, status information, etc.

[0075] The technical effects of the above technical solution are as follows: through preprocessing and category splitting, redundant data in the processing process is reduced and processing efficiency is improved. Distributed storage and parallel processing enable multiple service cluster nodes to process different data packets at the same time, further improving the processing speed. According to the performance evaluation results of the service cluster and the data packet, resources are reasonably allocated to avoid resource waste and bottleneck problems. Simulation processing service analysis helps to find potential problems before actual processing, so as to optimize and adjust in advance. Data cleaning and format conversion in the preprocessing step ensure the accuracy and consistency of the input data, which can improve the accuracy of the processing results. Multiple iterative processing and update processing can continuously correct and improve the processing results to make them more in line with actual needs. Distributed storage and parallel processing architecture enable the system to easily expand new service cluster nodes and data storage nodes to meet the growing processing needs. The system can flexibly adapt to different processing tasks and data types, improving the versatility and adaptability of the system.

[0076] In one embodiment of the present invention, the S1 includes:

[0077] Acquire electronic file processing request information, and acquire electronic file data that needs to be processed according to the electronic file processing request information;

[0078] Preprocessing the electronic file data to obtain preprocessed electronic file data;

[0079] Acquire preset electronic file splitting category information, and perform file data splitting on the preprocessed electronic file data according to the preset electronic file splitting category information to obtain a file splitting data packet;

[0080] Each file is split into data packets for distributed storage to obtain multiple distributed storage data packets.

[0081] The working principle of the above technical solution is as follows: receiving electronic file processing request information from a user or other system. The request information contains specific information of the electronic file to be processed, such as file type, file name, storage location, etc., as well as specific processing requirements, such as data cleaning, format conversion, splitting rules, etc. According to the electronic file processing request information, the system locates and obtains the corresponding electronic file data. These data are stored in a local file system, a network file system, a database or other storage medium. The obtained electronic file data is then input into a preprocessing module to perform necessary preprocessing operations such as data cleaning and format conversion to ensure the accuracy and consistency of the data. The preprocessed electronic file data is sent to a file splitting module. In this module, the system splits the file according to preset electronic file splitting category information. These splitting category information are defined based on the content, structure, size or other attributes of the file. The file splitting process involves cutting the file into multiple smaller parts, or reorganizing the file data into multiple data packets according to specific logical rules. The split data packets are usually easier to manage and process, especially in a distributed environment. Each file split data packet is then sent to a distributed storage system. A distributed storage system usually consists of multiple storage nodes, which may be distributed in different geographical locations or physical servers. In the distributed storage process, the system will distribute data packets to different storage nodes according to preset rules (such as load balancing, data redundancy, access speed, etc.). This ensures high availability and scalability of data while improving the speed and efficiency of data access.

[0082] The technical effect of the above technical solution is: through preprocessing and file splitting, the system can decompose complex electronic file processing tasks into multiple smaller and easier to manage tasks. This helps to speed up the processing process and improve overall processing efficiency. Distributed storage technology enables the system to make full use of the resources of multiple storage nodes to achieve load balancing and data redundancy. This can not only improve the reliability and security of data, but also optimize the utilization of storage resources and reduce storage costs. Distributed storage and file splitting technology enable the system to easily expand storage and processing capabilities. When more electronic files need to be processed or new processing functions need to be added, the system can be expanded by adding new storage nodes or processing nodes. The system can also flexibly adjust the splitting rules and storage strategies according to specific processing requirements to adapt to different application scenarios and user needs.

[0083] Distributed storage technology allows data to be redundantly stored on multiple storage nodes, thereby improving data reliability and availability. Even if a storage node fails, the system can still access data from other nodes to ensure business continuity.

[0084] In one embodiment of the present invention, S2 includes:

[0085] Acquire information of multiple preset service clusters, and calculate a data processing capacity coefficient of each preset service cluster according to the preset service cluster information;

[0086] The calculation formula of the data processing capacity coefficient is:

[0087]

[0088] Among them, JQX is the data processing capacity coefficient, j is the number of computer nodes in the preset service cluster, V lsi is the historical average data processing speed of the ith computer node, U lsi is the historical average processing success rate of the ith computer node, R lsi is the historical average resource utilization of the ith computer node, q is the number of preset service clusters, V lsin is the historical average data processing speed of the ith computer node in the nth preset service cluster, U lsin is the historical average processing success rate of the ith computer node in the nth preset service cluster, R lsin is the historical average resource utilization of the ith computer node of the nth preset service cluster;

[0089] According to the data processing capacity coefficient, the distributed storage data packets are allocated to the preset service cluster, such as Figure 2 As shown;

[0090] Obtaining a preset allocation data packet of a preset service cluster;

[0091] Performing simulation processing services on the preset allocated data packets through a preset service cluster to obtain cluster simulation processing data;

[0092] Calculating a cluster simulation processing coefficient according to the cluster simulation processing data;

[0093] The calculation formula of the cluster simulation processing coefficient is:

[0094]

[0095] Among them, JQM is the cluster simulation processing coefficient, j is the number of computer nodes in the preset service cluster, V sjbi is the simulated data processing speed of the preset allocated data packet of the i-th computer node, U sjbi is the simulated processing success rate of the preset allocated data packets of the i-th computer node, R sjbi is the simulated resource utilization of the preset allocated data packet of the i-th computer node, q is the number of preset service clusters, V sjbinThe simulated data processing speed of the preset allocated data packet of the ith computer node of the nth preset service cluster, U sjbin is the simulated processing success rate of the preset allocated data packet of the ith computer node of the nth preset service cluster, R sjbin The simulated resource utilization of the preset allocated data packet for the i-th computer node of the n-th preset service cluster;

[0096] A simulation processing determination is performed on the preset service cluster according to the cluster simulation processing coefficient to obtain simulation processing determination information of the preset service cluster.

[0097] The cluster simulation processing coefficient is compared with a preset simulation cluster coefficient threshold, and when the cluster simulation processing coefficient is greater than the preset simulation cluster coefficient threshold, the simulation processing is determined to be qualified processing.

[0098] The working principle of the above technical solution is: collect and obtain detailed information of multiple preset service clusters, including but not limited to the scale of the cluster (such as the number of nodes, hardware configuration), network bandwidth, storage capacity, current load, etc. This information is the basis for evaluating the data processing capability of each cluster. According to the preset service cluster information obtained, the data processing capability coefficient of each cluster is calculated. This coefficient reflects the ability of the cluster to process data in the current state. According to the calculated data processing capability coefficient, the system allocates the previously obtained multiple distributed storage data packets to each preset service cluster. The principle of allocation is to match the data processing load of each cluster with its capability as much as possible to achieve optimal utilization of resources. After receiving the allocated data packet, each preset service cluster uses its internal computing resources to simulate the data packet. This step is to evaluate the performance of the cluster when actually processing tasks. After the simulation processing is completed, the system calculates the simulation processing coefficient of each cluster based on the processing results. This coefficient reflects the efficiency and effect of the cluster when processing specific types of tasks. The preset service cluster is simulated and judged according to the cluster simulation processing coefficient. This judgment information includes whether the cluster is suitable for processing specific types of tasks, whether the cluster configuration needs to be adjusted to improve performance, etc.

[0099] The technical effect of the above technical solution is: by calculating and allocating the data processing capacity coefficient, the system can more accurately evaluate the resource status of each preset service cluster, and allocate data packet tasks accordingly, thereby avoiding resource overload or idleness and achieving optimal resource utilization. The simulation processing service can evaluate the performance of the cluster in actual task processing in advance, thereby helping the system to select the most suitable cluster to process the task and improve the overall processing efficiency. The system can dynamically adjust the task allocation strategy according to the simulation processing coefficient of the cluster, adapt to the changes in different scenarios and task requirements, and enhance the flexibility and adaptability of the system. Through reasonable resource allocation and task scheduling, the system can reduce unnecessary resource waste and reduce operating costs. At the same time, the simulation processing service can also help the system discover and solve potential performance bottlenecks in advance, avoiding serious problems in the actual operation process that lead to increased costs. The system can quickly respond to task requests according to the actual processing capacity of the cluster and provide stable and efficient services. The above calculation formula can be used to calculate the processing capacity coefficient and simulation processing coefficient of the preset service cluster affected by processing speed, processing success rate and resource utilization.

[0100] In one embodiment of the present invention, S3 includes:

[0101] Distributed storage data packet information, calculating a data packet processing capacity coefficient of each distributed storage data packet according to the distributed storage data packet information;

[0102] The calculation formula of the data packet processing capacity coefficient is:

[0103]

[0104] Among them, SJX is the data packet processing capacity coefficient, o is the number of distributed storage data packets, V lsbcl is the historical average data processing speed of distributed storage data packets, U lsbcl is the historical average successful processing rate of distributed storage data packets, R lsbcl is the historical average resource utilization of distributed storage data packets, V lsa is the historical average data processing speed of the ath distributed storage data packet, U lsa The ath is the historical average processing success rate of distributed storage data packets, R lsa is the historical average resource utilization rate of the a-th distributed storage data packet;

[0105] The distributed storage data packet is allocated a preset service cluster according to the data packet processing capacity coefficient, such as Figure 3 As shown;

[0106] Obtain a data packet cluster for distributed storage of data packets;

[0107] Performing simulation processing services on the distributed storage data packets through the data packet cluster to obtain data packet simulation processing data;

[0108] Calculating a data packet simulation processing coefficient according to the data packet simulation processing data;

[0109] The calculation formula of the data packet simulation processing coefficient is:

[0110]

[0111] Among them, SJM is the packet simulation processing coefficient, o is the number of distributed storage packets, V sjsjb is the processing speed of the simulated data of the distributed storage data packet, U sjsjb Simulates the processing success rate of distributed storage data packets, R sjsjb is the resource utilization of the simulated distributed storage data packets, V sja is the processing speed of the simulated data of the ath distributed storage data packet, U sja is the simulated processing success rate of the ath distributed storage data packet, R sja is the simulated resource utilization of the a-th distributed storage data packet;

[0112] The distributed storage data packet is subjected to simulation processing determination according to the data packet simulation processing coefficient to obtain simulation processing determination information of the distributed storage data packet.

[0113] The working principle of the above technical solution is: receiving or obtaining information of multiple distributed storage data packets, including but not limited to the size, data type, data importance level, data access frequency, etc. of the data packets. Based on the obtained distributed storage data packet information, the processing capacity coefficient of each data packet is calculated. This coefficient reflects the difficulty of the data packet in the processing process and the amount of resources required. According to the calculated data packet processing capacity coefficient, the system allocates the distributed storage data packet to a suitable preset service cluster. The principle of allocation is to match the processing requirements of the data packet with the processing capacity of the cluster as much as possible to achieve optimal resource utilization and the highest processing efficiency. The system confirms the specific cluster to which each data packet is allocated, that is, the data packet cluster. This is the basis for subsequent simulation processing services. The data packets received by each data packet cluster will be simulated. The simulation processing is to evaluate the performance of the data packet in the actual processing process, including processing speed, resource consumption, etc. After the simulation processing is completed, the system calculates the simulation processing coefficient of each data packet according to the processing results. This coefficient reflects the efficiency and effect of the data packet processing on a specific cluster. The distributed storage data packet is simulated and judged according to the data packet simulation processing coefficient. This judgment information may include whether the data packet is suitable for processing on the current cluster, whether the processing strategy needs to be adjusted or the cluster configuration needs to be optimized, etc.

[0114] The technical effect of the above technical solution is: by calculating the processing capacity coefficient of the data packet and allocating the preset service cluster accordingly, the system can more accurately evaluate the processing requirements of each data packet relative to the processing capacity of the cluster, and then obtain the processing difficulty of each data packet. The simulation processing service can evaluate the performance of the data packet in the actual processing process in advance, help the system select the most appropriate cluster and processing strategy, thereby improving the overall processing efficiency. The system can dynamically adjust the processing strategy according to the simulation processing coefficient of the data packet, adapt to the changes in different data packet types and processing requirements, and enhance the flexibility and adaptability of the system. Through reasonable resource allocation and task scheduling, the system can reduce unnecessary resource waste and reduce operating costs. At the same time, the simulation processing service can also help the system discover and solve potential performance bottlenecks in advance, avoiding serious problems in the actual operation process that lead to increased costs. The system can quickly respond to the processing request of the data packet, provide stable and efficient services, and thus improve the user experience. Especially when processing big data and complex tasks, the optimized configuration and efficient processing of the system can significantly improve user satisfaction and trust.

[0115] In one embodiment of the present invention, the S4 includes:

[0116] Sort the preset service clusters from large to small by cluster simulation processing coefficients to obtain cluster processing ranking;

[0117] The distributed storage data packets are sorted from large to small by using the data packet simulation processing coefficient to obtain the data packet sorting;

[0118] The distributed storage data packets are actually processed through preset service clusters of equal ranking to obtain actual processing data;

[0119] Compare the actual processing data with the preset processing threshold value to obtain a processing comparison result;

[0120] The cluster processing order and the data packet order are updated according to the processing comparison result, and then actual reprocessing is performed to obtain a reprocessing result.

[0121] The working principle of the above technical solution is as follows: the processing capacity of the preset service cluster is evaluated, and the simulated processing capacity of each cluster is measured by the simulated processing coefficient. According to these processing coefficients, the preset service clusters are sorted from large to small to obtain a cluster processing sorting list. The data packets in the distributed storage are also evaluated, and the simulated processing capacity of each data packet is measured by the simulated processing coefficient. According to these processing coefficients, the data packets are sorted from large to small to obtain a data packet sorting list. According to the correspondence between the cluster processing sorting and the data packet sorting, the preset service clusters with similar (or equal) rankings are assigned to the corresponding data packets for actual processing. After the processing is completed, the actual processing data is collected and stored, which reflects the actual effect of the cluster processing the data packet. The system compares the actual processing data with the preset processing threshold to evaluate whether the processing effect meets expectations. According to the comparison results, the system dynamically updates the cluster processing sorting and the data packet sorting. For example, if a cluster has poor processing effect, its position in the sorting may be lowered; if the processing demand of a data packet is underestimated, its sorting position may be increased. The updated sorting list will be used to guide the subsequent actual reprocessing process. Based on the updated sorted list, the system performs actual processing again to obtain more accurate and optimized processing results.

[0122] The technical effect of the above technical solution is: through precise sorting and matching, the system can allocate resources more reasonably, ensuring that clusters with strong processing capabilities can process data packets with high demand, thereby improving overall resource utilization. By dynamically updating the sorting list, the system can adjust the processing strategy according to actual conditions, avoid resource waste and bottleneck problems, and thus improve processing efficiency. The system can adjust the sorting in real time according to the processing comparison results to adapt to different processing requirements and scenario changes, enhancing the flexibility and adaptability of the system. By optimizing the processing flow and improving processing efficiency, the system can respond to user requests faster and provide a more stable and efficient service experience. The automated and intelligent processing flow reduces the need for manual intervention and reduces operation and maintenance costs and time costs.

[0123] In one embodiment of the present invention, the processing system comprises:

[0124] A file data splitting module is used to obtain electronic file data, pre-process and classify the electronic file data, obtain file split data packets, and then obtain distributed storage data packets;

[0125] The cluster capability analysis module is used to analyze the data processing capability of the acquired preset service cluster, allocate distributed storage data packets according to the evaluation results, perform simulated processing service analysis on the preset allocated data packets, and then perform simulated processing judgment on the preset service cluster;

[0126] The data packet capability analysis module is used to analyze the processing capability of distributed storage data packets, allocate preset service clusters to distributed storage data packets, perform simulated processing service analysis, and then perform simulated processing judgment on distributed storage data packets;

[0127] The actual processing module is used to obtain cluster processing ranking and data packet ranking, and to perform actual processing and update processing on distributed storage data packets through preset service clusters with the same ranking to obtain processing results.

[0128] The working principle of the above technical solution is as follows: electronic file data is obtained from various sources (such as databases, file servers, networks, etc.). The electronic file data is preprocessed, including data cleaning (removing irrelevant information, correcting erroneous data, etc.), format conversion (unifying file formats), category splitting (dividing files into different categories according to their content or attributes), etc., to obtain file split data packets. The file split data packets are further processed into distributed storage data packets for storage and access on multiple storage nodes. The performance of data packet processing is evaluated for a preset service cluster (composed of multiple computing nodes). According to the evaluation results, data packets are allocated to service cluster nodes. Simulated processing service analysis is performed on the preset allocated data packets, and the preliminary processing capacity of the service cluster is evaluated by simulating the running of processing tasks, and a simulated processing judgment is made. The processing capacity of the distributed storage data packets is analyzed, that is, the impact of factors such as the size and complexity of the data packets on the processing capacity is evaluated. According to the analysis results, the preset service cluster nodes are allocated to the data packets, and the simulated data processing capacity of the data packets is evaluated. Simulated processing service analysis is performed on the allocated data packets, and a simulated processing judgment is made. Get the cluster processing ranking and data packet ranking, and select the preset service cluster with the same ranking to actually process the distributed storage data packet according to the ranking results. During the processing, the data packet needs to be iterated or updated multiple times to obtain the final processing result. The processing result may include data analysis reports, processed data files, status information, etc.

[0129] The technical effects of the above technical solution are as follows: through preprocessing and category splitting, redundant data in the processing process is reduced and processing efficiency is improved. Distributed storage and parallel processing enable multiple service cluster nodes to process different data packets at the same time, further improving the processing speed. According to the performance evaluation results of the service cluster and the data packet, resources are reasonably allocated to avoid resource waste and bottleneck problems. Simulation processing service analysis helps to find potential problems before actual processing, so as to optimize and adjust in advance. Data cleaning and format conversion in the preprocessing step ensure the accuracy and consistency of the input data, which can improve the accuracy of the processing results. Multiple iterative processing and update processing can continuously correct and improve the processing results to make them more in line with actual needs. Distributed storage and parallel processing architecture enable the system to easily expand new service cluster nodes and data storage nodes to meet the growing processing needs. The system can flexibly adapt to different processing tasks and data types, improving the versatility and adaptability of the system.

[0130] In one embodiment of the present invention, the file data splitting module includes:

[0131] Acquiring electronic file processing request information, and acquiring electronic file data that needs to be processed according to the electronic file processing request information;

[0132] Preprocessing the electronic file data to obtain preprocessed electronic file data;

[0133] Acquire preset electronic file splitting category information, and perform file data splitting on the preprocessed electronic file data according to the preset electronic file splitting category information to obtain a file splitting data packet;

[0134] Each file is split into data packets for distributed storage to obtain multiple distributed storage data packets.

[0135] The working principle of the above technical solution is as follows: receiving electronic file processing request information from a user or other system. The request information contains specific information of the electronic file to be processed, such as file type, file name, storage location, etc., as well as specific processing requirements, such as data cleaning, format conversion, splitting rules, etc. According to the electronic file processing request information, the system locates and obtains the corresponding electronic file data. These data are stored in a local file system, a network file system, a database or other storage medium. The obtained electronic file data is then input into a preprocessing module to perform necessary preprocessing operations such as data cleaning and format conversion to ensure the accuracy and consistency of the data. The preprocessed electronic file data is sent to a file splitting module. In this module, the system splits the file according to preset electronic file splitting category information. These splitting category information are defined based on the content, structure, size or other attributes of the file. The file splitting process involves cutting the file into multiple smaller parts, or reorganizing the file data into multiple data packets according to specific logical rules. The split data packets are usually easier to manage and process, especially in a distributed environment. Each file split data packet is then sent to a distributed storage system. A distributed storage system usually consists of multiple storage nodes, which may be distributed in different geographical locations or physical servers. In the distributed storage process, the system will distribute data packets to different storage nodes according to preset rules (such as load balancing, data redundancy, access speed, etc.). This ensures high availability and scalability of data while improving the speed and efficiency of data access.

[0136] The technical effect of the above technical solution is: through preprocessing and file splitting, the system can decompose complex electronic file processing tasks into multiple smaller and easier to manage tasks. This helps to speed up the processing process and improve overall processing efficiency. Distributed storage technology enables the system to make full use of the resources of multiple storage nodes to achieve load balancing and data redundancy. This can not only improve the reliability and security of data, but also optimize the utilization of storage resources and reduce storage costs. Distributed storage and file splitting technology enable the system to easily expand storage and processing capabilities. When more electronic files need to be processed or new processing functions need to be added, the system can be expanded by adding new storage nodes or processing nodes. The system can also flexibly adjust the splitting rules and storage strategies according to specific processing requirements to adapt to different application scenarios and user needs.

[0137] Distributed storage technology allows data to be redundantly stored on multiple storage nodes, thereby improving data reliability and availability. Even if a storage node fails, the system can still access data from other nodes to ensure business continuity.

[0138] In one embodiment of the present invention, the cluster capability analysis module includes:

[0139] Acquire information of multiple preset service clusters, and calculate a data processing capacity coefficient of each preset service cluster according to the preset service cluster information;

[0140] The calculation formula of the data processing capacity coefficient is:

[0141]

[0142] Among them, JQX is the data processing capacity coefficient, j is the number of computer nodes in the preset service cluster, V lsi is the historical average data processing speed of the ith computer node, U lsi is the historical average processing success rate of the ith computer node, R lsi is the historical average resource utilization of the ith computer node, q is the number of preset service clusters, V lsin is the historical average data processing speed of the ith computer node in the nth preset service cluster, U lsin is the historical average processing success rate of the ith computer node in the nth preset service cluster, R lsin is the historical average resource utilization of the ith computer node of the nth preset service cluster;

[0143] Allocating distributed storage data packets to the preset service cluster according to the data processing capacity coefficient;

[0144] Obtaining a preset allocation data packet of a preset service cluster;

[0145] Performing simulation processing services on the preset allocated data packets through a preset service cluster to obtain cluster simulation processing data;

[0146] Calculating a cluster simulation processing coefficient according to the cluster simulation processing data;

[0147] The calculation formula of the cluster simulation processing coefficient is:

[0148]

[0149] Among them, JQM is the cluster simulation processing coefficient, j is the number of computer nodes in the preset service cluster, V sjbi is the data processing speed of the preset allocated data packet of the i-th computer node, U sjbi is the processing success rate of the preset allocated data packets of the i-th computer node, R sjbi is the resource utilization rate of the preset allocated data packet of the i-th computer node, q is the number of preset service clusters, V sjbin The data processing speed of the preset allocated data packet of the ith computer node of the nth preset service cluster, U sjbinis the processing success rate of the preset allocated data packet of the ith computer node of the nth preset service cluster, R sjbin resource utilization of a preset allocated data packet for an i-th computer node of an n-th preset service cluster;

[0150] A simulation processing determination is performed on the preset service cluster according to the cluster simulation processing coefficient to obtain simulation processing determination information of the preset service cluster.

[0151] The working principle of the above technical solution is: collect and obtain detailed information of multiple preset service clusters, including but not limited to the scale of the cluster (such as the number of nodes, hardware configuration), network bandwidth, storage capacity, current load, etc. This information is the basis for evaluating the data processing capability of each cluster. According to the preset service cluster information obtained, the data processing capability coefficient of each cluster is calculated. This coefficient reflects the ability of the cluster to process data in the current state. According to the calculated data processing capability coefficient, the system allocates the previously obtained multiple distributed storage data packets to each preset service cluster. The principle of allocation is to match the data processing load of each cluster with its capability as much as possible to achieve optimal utilization of resources. After receiving the allocated data packet, each preset service cluster uses its internal computing resources to simulate the data packet. This step is to evaluate the performance of the cluster when actually processing tasks. After the simulation processing is completed, the system calculates the simulation processing coefficient of each cluster based on the processing results. This coefficient reflects the efficiency and effect of the cluster when processing specific types of tasks. The preset service cluster is simulated and judged according to the cluster simulation processing coefficient. This judgment information includes whether the cluster is suitable for processing specific types of tasks, whether the cluster configuration needs to be adjusted to improve performance, etc.

[0152] The technical effect of the above technical solution is: by calculating and allocating the data processing capacity coefficient, the system can more accurately evaluate the resource status of each preset service cluster, and allocate data packet tasks accordingly, so as to avoid resource overload or idleness and achieve optimal utilization of resources. The simulation processing service can evaluate the performance of the cluster in actual task processing in advance, so as to help the system select the most suitable cluster to process the task and improve the overall processing efficiency. The system can dynamically adjust the task allocation strategy according to the simulation processing coefficient of the cluster, adapt to the changes in different scenarios and task requirements, and enhance the flexibility and adaptability of the system. Through reasonable resource allocation and task scheduling, the system can reduce unnecessary waste of resources and reduce operating costs. At the same time, the simulation processing service can also help the system discover and solve potential performance bottlenecks in advance, avoiding serious problems in the actual operation process that lead to increased costs. The system can quickly respond to task requests according to the actual processing capacity of the cluster and provide stable and efficient services.

[0153] In one embodiment of the present invention, the data packet capability analysis module includes:

[0154] Distributed storage data packet information, calculating a data packet processing capacity coefficient of each distributed storage data packet according to the distributed storage data packet information;

[0155] The calculation formula of the data packet processing capacity coefficient is:

[0156]

[0157] Among them, SJX is the data packet processing capacity coefficient, o is the number of distributed storage data packets, V lsbcl is the historical average data processing speed of distributed storage data packets, U lsbcl is the historical average successful processing rate of distributed storage data packets, R lsbcl is the historical average resource utilization of distributed storage data packets, V lsa is the historical average data processing speed of the ath distributed storage data packet, U lsa The ath is the historical average processing success rate of distributed storage data packets, R lsa is the historical average resource utilization rate of the a-th distributed storage data packet;

[0158] Allocating a preset service cluster to the distributed storage data packet according to the data packet processing capacity coefficient;

[0159] Obtain a data packet cluster for distributed storage of data packets;

[0160] Performing simulation processing services on the distributed storage data packets through the data packet cluster to obtain data packet simulation processing data;

[0161] Calculating a data packet simulation processing coefficient according to the data packet simulation processing data;

[0162] The calculation formula of the data packet simulation processing coefficient is:

[0163]

[0164] Among them, SJM is the packet simulation processing coefficient, o is the number of distributed storage packets, V sjsjb is the processing speed of the simulated data of the distributed storage data packet, U sjsjb Simulates the processing success rate of distributed storage data packets, R sjsjb is the resource utilization of the simulated distributed storage data packets, V sja is the processing speed of the simulated data of the ath distributed storage data packet, U sja is the simulated processing success rate of the ath distributed storage data packet, Rsja is the simulated resource utilization of the a-th distributed storage data packet;

[0165] The distributed storage data packet is subjected to simulation processing determination according to the data packet simulation processing coefficient to obtain simulation processing determination information of the distributed storage data packet.

[0166] The working principle of the above technical solution is: receiving or obtaining information of multiple distributed storage data packets, including but not limited to the size, data type, data importance level, data access frequency, etc. of the data packets. Based on the obtained distributed storage data packet information, the processing capacity coefficient of each data packet is calculated. This coefficient reflects the difficulty of the data packet in the processing process and the amount of resources required. According to the calculated data packet processing capacity coefficient, the system allocates the distributed storage data packet to a suitable preset service cluster. The principle of allocation is to match the processing requirements of the data packet with the processing capacity of the cluster as much as possible to achieve optimal resource utilization and the highest processing efficiency. The system confirms the specific cluster to which each data packet is allocated, that is, the data packet cluster. This is the basis for subsequent simulation processing services. The data packets received by each data packet cluster will be simulated. The simulation processing is to evaluate the performance of the data packet in the actual processing process, including processing speed, resource consumption, etc. After the simulation processing is completed, the system calculates the simulation processing coefficient of each data packet according to the processing results. This coefficient reflects the efficiency and effect of the data packet processing on a specific cluster. The distributed storage data packet is simulated and judged according to the data packet simulation processing coefficient. This judgment information may include whether the data packet is suitable for processing on the current cluster, whether the processing strategy needs to be adjusted or the cluster configuration needs to be optimized, etc.

[0167] The technical effect of the above technical solution is: by calculating the processing capacity coefficient of the data packet and allocating the preset service cluster accordingly, the system can more accurately evaluate the processing requirements of each data packet and the processing capacity of each cluster, thereby realizing the optimal configuration and efficient utilization of resources. The simulation processing service can evaluate the performance of the data packet in the actual processing process in advance, help the system select the most appropriate cluster and processing strategy, thereby improving the overall processing efficiency. The system can dynamically adjust the processing strategy according to the simulation processing coefficient of the data packet, adapt to the changes of different data packet types and processing requirements, and enhance the flexibility and adaptability of the system. Through reasonable resource allocation and task scheduling, the system can reduce unnecessary resource waste and reduce operating costs. At the same time, the simulation processing service can also help the system discover and solve potential performance bottlenecks in advance, avoiding serious problems in the actual operation process that lead to increased costs. The system can quickly respond to the processing request of the data packet and provide stable and efficient services, thereby improving the user experience. Especially when processing big data and complex tasks, the optimized configuration and efficient processing of the system can significantly improve user satisfaction and trust.

[0168] In one embodiment of the present invention, the actual processing module includes:

[0169] Sort the preset service clusters from large to small by cluster simulation processing coefficients to obtain cluster processing ranking;

[0170] The distributed storage data packets are sorted from large to small by using the data packet simulation processing coefficient to obtain the data packet sorting;

[0171] The distributed storage data packets are actually processed through preset service clusters of equal ranking to obtain actual processing data;

[0172] Compare the actual processing data with the preset processing threshold value to obtain a processing comparison result;

[0173] The cluster processing order and the data packet order are updated according to the processing comparison result, and then actual reprocessing is performed to obtain a reprocessing result.

[0174] The working principle of the above technical solution is as follows: the processing capacity of the preset service cluster is evaluated, and the simulated processing capacity of each cluster is measured by the simulated processing coefficient. According to these processing coefficients, the preset service clusters are sorted from large to small to obtain a cluster processing sorting list. The data packets in the distributed storage are also evaluated, and the simulated processing capacity of each data packet is measured by the simulated processing coefficient. According to these processing coefficients, the data packets are sorted from large to small to obtain a data packet sorting list. According to the correspondence between the cluster processing sorting and the data packet sorting, the preset service clusters with similar (or equal) rankings are assigned to the corresponding data packets for actual processing. After the processing is completed, the actual processing data is collected and stored, which reflects the actual effect of the cluster processing the data packet. The system compares the actual processing data with the preset processing threshold to evaluate whether the processing effect meets expectations. According to the comparison results, the system dynamically updates the cluster processing sorting and the data packet sorting. For example, if a cluster has poor processing effect, its position in the sorting may be lowered; if the processing demand of a data packet is underestimated, its sorting position may be increased. The updated sorting list will be used to guide the subsequent actual reprocessing process. Based on the updated sorted list, the system performs actual processing again to obtain more accurate and optimized processing results.

[0175] The technical effect of the above technical solution is: through precise sorting and matching, the system can allocate resources more reasonably, ensuring that clusters with strong processing capabilities can process data packets with high demand, thereby improving overall resource utilization. By dynamically updating the sorting list, the system can adjust the processing strategy according to actual conditions, avoid resource waste and bottleneck problems, and thus improve processing efficiency. The system can adjust the sorting in real time according to the processing comparison results to adapt to different processing requirements and scenario changes, enhancing the flexibility and adaptability of the system. By optimizing the processing flow and improving processing efficiency, the system can respond to user requests faster and provide a more stable and efficient service experience. The automated and intelligent processing flow reduces the need for manual intervention and reduces operation and maintenance costs and time costs.

[0176] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for processing electronic file data based on big data, characterized in that: The processing method comprises: S1. Acquire electronic file data, pre-process and split the electronic file data into categories, obtain file split data packets, and then obtain distributed storage data packets; S2, analyzing the data processing capability of the acquired preset service cluster, allocating distributed storage data packets according to the evaluation results, performing simulated processing service analysis on the preset allocated data packets, and then performing simulated processing judgment on the preset service cluster; Wherein, the S2 includes: Acquire information of multiple preset service clusters, and calculate a data processing capacity coefficient of each preset service cluster according to the preset service cluster information; Allocating distributed storage data packets to the preset service cluster according to the data processing capacity coefficient; Obtaining a preset allocation data packet of a preset service cluster; Performing simulation processing services on the preset allocated data packets through a preset service cluster to obtain cluster simulation processing data; Calculating a cluster simulation processing coefficient according to the cluster simulation processing data; The calculation formula of the cluster simulation processing coefficient is: Among them, JQM is the cluster simulation processing coefficient, j is the number of computer nodes in the preset service cluster, V sjbi is the simulated data processing speed of the preset allocated data packet of the i-th computer node, U sjbi is the simulated processing success rate of the preset allocated data packets of the i-th computer node, R sjbi is the simulated resource utilization of the preset allocated data packet of the i-th computer node, q is the number of preset service clusters, V sjbin The simulated data processing speed of the preset allocated data packet of the ith computer node of the nth preset service cluster, U sjbin is the simulated processing success rate of the preset allocated data packet of the ith computer node of the nth preset service cluster, R sjbin The simulated resource utilization of the preset allocated data packet for the i-th computer node of the n-th preset service cluster; Performing simulation processing determination on the preset service cluster according to the cluster simulation processing coefficient to obtain simulation processing determination information of the preset service cluster; S3, analyzing the processing capacity of the distributed storage data packets, assigning preset service clusters to the distributed storage data packets, performing simulation processing service analysis, and then performing simulation processing determination on the distributed storage data packets; S4. Obtain cluster processing ranking and data packet ranking, and perform actual processing and update processing on the distributed storage data packet through the preset service cluster with the same ranking to obtain the processing result.

2. The method for processing electronic file data based on big data according to claim 1, characterized in that: The S1 includes: Acquiring electronic file processing request information, and acquiring electronic file data that needs to be processed according to the electronic file processing request information; Preprocessing the electronic file data to obtain preprocessed electronic file data; Acquire preset electronic file splitting category information, and perform file data splitting on the preprocessed electronic file data according to the preset electronic file splitting category information to obtain a file splitting data packet; Each file is split into data packets for distributed storage to obtain multiple distributed storage data packets.

3. The method for processing electronic file data based on big data according to claim 1, characterized in that: The S3 includes: Distributed storage data packet information, calculating a data packet processing capacity coefficient of each distributed storage data packet according to the distributed storage data packet information; Allocating a preset service cluster to the distributed storage data packet according to the data packet processing capacity coefficient; Obtain a data packet cluster for distributed storage of data packets; Performing simulation processing services on the distributed storage data packets through the data packet cluster to obtain data packet simulation processing data; Calculating a data packet simulation processing coefficient according to the data packet simulation processing data; The calculation formula of the data packet simulation processing coefficient is: Among them, SJM is the packet simulation processing coefficient, o is the number of distributed storage packets, V sjsjb is the processing speed of the simulated data of the distributed storage data packet, U sjsjb Simulates the processing success rate of distributed storage data packets, R sjsjb is the resource utilization of the simulated distributed storage data packets, V sja is the processing speed of the simulated data of the ath distributed storage data packet, U sja is the simulated processing success rate of the ath distributed storage data packet, R sja is the simulated resource utilization of the a-th distributed storage data packet; The distributed storage data packet is subjected to simulation processing determination according to the data packet simulation processing coefficient to obtain simulation processing determination information of the distributed storage data packet.

4. The method for processing electronic file data based on big data according to claim 3, characterized in that: The S4 includes: Sort the preset service clusters from large to small by cluster simulation processing coefficients to obtain cluster processing ranking; The distributed storage data packets are sorted from large to small by using the data packet simulation processing coefficient to obtain the data packet sorting; The distributed storage data packets are actually processed through preset service clusters of equal ranking to obtain actual processing data; Compare the actual processing data with the preset processing threshold value to obtain a processing comparison result; The cluster processing order and the data packet order are updated according to the processing comparison result, and then actual reprocessing is performed to obtain a reprocessing result.

5. An electronic document data processing system based on big data, characterized in that: The processing system comprises: A file data splitting module is used to obtain electronic file data, pre-process and classify the electronic file data, obtain file split data packets, and then obtain distributed storage data packets; The cluster capability analysis module is used to analyze the data processing capability of the acquired preset service cluster, allocate distributed storage data packets according to the evaluation results, perform simulated processing service analysis on the preset allocated data packets, and then perform simulated processing judgment on the preset service cluster; Wherein, the cluster capability analysis module includes: Acquire information of multiple preset service clusters, and calculate a data processing capacity coefficient of each preset service cluster according to the preset service cluster information; Allocating distributed storage data packets to the preset service cluster according to the data processing capacity coefficient; Obtaining a preset allocation data packet of a preset service cluster; Performing simulation processing services on the preset allocated data packets through a preset service cluster to obtain cluster simulation processing data; Calculating a cluster simulation processing coefficient according to the cluster simulation processing data; The calculation formula of the cluster simulation processing coefficient is: Among them, JQM is the cluster simulation processing coefficient, j is the number of computer nodes in the preset service cluster, V sjbi is the simulated data processing speed of the preset allocated data packet of the i-th computer node, U sjbi is the simulated processing success rate of the preset allocated data packets of the i-th computer node, R sjbi is the simulated resource utilization of the preset allocated data packet of the i-th computer node, q is the number of preset service clusters, V sjbin The simulated data processing speed of the preset allocated data packet of the ith computer node of the nth preset service cluster, U sjbin is the simulated processing success rate of the preset allocated data packet of the ith computer node of the nth preset service cluster, R sjbin The simulated resource utilization of the preset allocated data packet for the i-th computer node of the n-th preset service cluster; Performing simulation processing determination on the preset service cluster according to the cluster simulation processing coefficient to obtain simulation processing determination information of the preset service cluster; The data packet capability analysis module is used to analyze the processing capability of distributed storage data packets, allocate preset service clusters to distributed storage data packets, perform simulated processing service analysis, and then perform simulated processing judgment on distributed storage data packets; The actual processing module is used to obtain cluster processing ranking and data packet ranking, and to perform actual processing and update processing on distributed storage data packets through preset service clusters with the same ranking to obtain processing results.

6. The electronic document data processing system based on big data according to claim 5, characterized in that: The file data splitting module includes: Acquiring electronic file processing request information, and acquiring electronic file data that needs to be processed according to the electronic file processing request information; Preprocessing the electronic file data to obtain preprocessed electronic file data; Acquire preset electronic file splitting category information, and perform file data splitting on the preprocessed electronic file data according to the preset electronic file splitting category information to obtain a file splitting data packet; Each file is split into data packets for distributed storage to obtain multiple distributed storage data packets.

7. The electronic document data processing system based on big data according to claim 5 is characterized in that: The data packet capability analysis module comprises: Distributed storage data packet information, calculating a data packet processing capacity coefficient of each distributed storage data packet according to the distributed storage data packet information; Allocating a preset service cluster to the distributed storage data packet according to the data packet processing capacity coefficient; Obtain a data packet cluster for distributed storage of data packets; Performing simulation processing services on the distributed storage data packets through the data packet cluster to obtain data packet simulation processing data; Calculating a data packet simulation processing coefficient according to the data packet simulation processing data; The calculation formula of the data packet simulation processing coefficient is: Among them, SJM is the packet simulation processing coefficient, o is the number of distributed storage packets, V sjsjb is the processing speed of the simulated data of the distributed storage data packet, U sjsjb Simulates the processing success rate of distributed storage data packets, R sjsjb is the resource utilization of the simulated distributed storage data packets, V sja is the processing speed of the simulated data of the ath distributed storage data packet, U sja is the simulated processing success rate of the ath distributed storage data packet, R sja is the simulated resource utilization of the a-th distributed storage data packet; The distributed storage data packet is subjected to simulation processing determination according to the data packet simulation processing coefficient to obtain simulation processing determination information of the distributed storage data packet.

8. The electronic document data processing system based on big data according to claim 7 is characterized in that: The actual processing module includes: Sort the preset service clusters from large to small by cluster simulation processing coefficients to obtain cluster processing ranking; The distributed storage data packets are sorted from large to small by using the data packet simulation processing coefficient to obtain the data packet sorting; The distributed storage data packets are actually processed through preset service clusters of equal ranking to obtain actual processing data; Compare the actual processing data with the preset processing threshold value to obtain a processing comparison result; The cluster processing order and the data packet order are updated according to the processing comparison result, and then actual reprocessing is performed to obtain a reprocessing result.

Images