Distributed file processing method and system

Abstract

The embodiment of the invention discloses a distributed file processing method and system.The method comprises the steps that connection with child nodes is established, and load information sent by the child nodes is received regularly; when a to-be-processed file is received, the to-be-processed file is fragmented, and a serial number is allocated to each sub-file; calculating the number of distributable sub-files of each sub-node according to the load information, and distributing the sub-files to the corresponding sub-nodes according to the number of distributable sub-files; adjusting the number of sub-nodes and the distribution of sub-files regularly according to the load information; and receiving execution results of the sub-nodes on the sub-files, sorting the execution results of all the sub-files according to the serial numbers, and splicing the sorted execution results to obtain a final execution result of the to-be-processed file.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and particularly to a distributed file processing method and system. Background Art

[0002] In the current data processing environment, with the rapid growth of data volume, the traditional single-machine file compression method faces problems of poor scalability and low efficiency. Especially when facing large-scale data sets, a single server may not be able to meet the requirements of fast compression and decompression. In addition, existing distributed compression schemes often have deficiencies such as unreasonable resource allocation and weak fault recovery capabilities.

[0003] For the above problems in the prior art, there is currently no effective solution. Summary of the Invention

[0004] To solve the above problems, the present invention provides a distributed file processing method and system. By analyzing the load conditions of each sub-node, the sharded sub-files are intelligently allocated to each sub-node to ensure reasonable resource allocation and solve the problems of unreasonable resource allocation and low processing efficiency in the prior art.

[0005] To achieve the above object, the present invention provides a distributed file processing method, including: establishing a connection with the sub-nodes and periodically receiving the load information sent by the sub-nodes; when receiving a file to be processed, sharding the file to be processed and assigning a serial number to each sub-file; calculating the number of sub-files that can be allocated to each sub-node according to the load information, and allocating sub-files to the corresponding sub-nodes according to the number of sub-files that can be allocated; periodically adjusting the number of sub-nodes and the allocation of sub-files according to the load information; receiving the execution results of the sub-files by the sub-nodes, sorting and splicing the execution results of all sub-files according to the serial numbers to obtain the final execution result of the file to be processed.

[0006] Further optionally, the calculating the number of sub-files that can be allocated to each sub-node according to the load information includes: calculating the real-time load of the sub-node according to the load information; calculating the average load according to the real-time load and the number of sub-nodes; selecting the sub-nodes with real-time load lower than the average load, and calculating the number of sub-files that can be allocated to the corresponding sub-nodes according to the real-time load and the average load.

[0007] Further optionally, the calculating the real-time load of the sub-node according to the load information includes: performing a weighted sum according to the CPU usage rate, memory usage rate, disk I / O usage rate, network bandwidth usage rate and the number of tasks in the load information to obtain the real-time load;

[0008] The calculating the number of sub-files that can be allocated to the corresponding sub-node according to the real-time load and the average load is calculated by the following formula:

[0009]

[0010] wherein, T i represents the number of allocable sub-files corresponding to the i-th child node; N represents the total number of sub-files; Load i represents the real-time load of the current computing child node i, and Load ave is the average load, and Load j represents the real-time load of the j-th child node, and n represents the total number of child nodes.

[0011] Further optionally, the timing to adjust the number of child nodes and the allocation of sub-files according to the load information includes: calculating the real-time load and average load of the child nodes according to the load information; when the average load is greater than a preset high load threshold and the duration is greater than a first preset time threshold, establishing a connection with a standby node, recalculating the number of allocable sub-files of each child node according to the load information, and allocating sub-files to the corresponding child nodes according to the number of allocable sub-files; when the average load is lower than a preset low load threshold and the duration is greater than a second preset time threshold, identifying low-load child nodes according to the real-time load, and migrating the sub-files of the low-load child nodes to other child nodes.

[0012] Further optionally, the establishing a connection with a child node includes: when receiving a connection request from a child node, sending the public key in the asymmetric key pair to the child node; wherein, the asymmetric key pair is pre-generated; receiving a registration request sent by the child node, and decrypting the symmetric encryption key and node information in the registration request through the private key; wherein, the symmetric encryption key and node information in the registration request are pre-encrypted by the child node through the public key; registering on the local side according to the node information, associating the symmetric encryption key with the corresponding child node, and sending a registration success notification to the child node.

[0013] On the other hand, the present invention also provides a distributed file processing system, including: a connection module, configured to establish a connection with a child node and periodically receive load information sent by the child node; a sharding module, configured to shard a file to be processed when receiving the file to be processed, and assign a serial number to each sharded sub-file; a task allocation module, configured to calculate the number of allocable sub-files of each child node according to the load information, and allocate sub-files to the corresponding child nodes according to the number of allocable sub-files; a dynamic adjustment module, configured to periodically adjust the number of child nodes and the allocation of sub-files according to the load information; a splicing module, configured to receive the execution results of the sub-files from the child nodes, sort and splice the execution results of all sub-files according to the serial numbers to obtain the final execution result of the file to be processed.

[0014] Further optionally, the task allocation module includes: a real-time load calculation sub-module, configured to calculate the real-time load of the sub-nodes according to the load information; an average load calculation sub-module, configured to calculate the average load according to the real-time load and the number of sub-nodes; a quantity calculation sub-module, configured to select the sub-nodes with a real-time load lower than the average load, and calculate the number of distributable sub-files corresponding to the sub-nodes according to the real-time load and the average load.

[0015] Further optionally, calculating the real-time load of the sub-nodes according to the load information includes: performing a weighted sum according to the CPU usage rate, memory usage rate, disk I / O usage rate, network bandwidth usage rate and the number of tasks in the load information to obtain the real-time load;

[0016] Calculating the number of distributable sub-files corresponding to the sub-nodes according to the real-time load and the average load is calculated by the following formula:

[0017]

[0018] where T i represents the number of distributable sub-files corresponding to the i-th sub-node; N represents the total number of sub-files; Load i represents the real-time load of the currently calculated sub-node i, Load ave is the average load, Load j represents the real-time load of the j-th sub-node, and n represents the total number of sub-nodes.

[0019] Further optionally, the dynamic adjustment module includes: a load calculation sub-module, configured to calculate the real-time load and the average load of the sub-nodes according to the load information; a node addition sub-module, configured to establish a connection with a standby node when the average load is greater than a preset high load threshold and the duration is greater than a first preset time threshold, recalculate the number of distributable sub-files of each sub-node according to the load information, and allocate sub-files to the corresponding sub-nodes according to the number of distributable sub-files; a node reduction sub-module, configured to identify low-load sub-nodes according to the real-time load when the average load is lower than a preset low load threshold and the duration is greater than a second preset time threshold, and migrate the sub-files of the low-load sub-nodes to other sub-nodes.

[0020] Further optionally, the connection module includes: a public key sending sub-module, configured to send the public key in the asymmetric key pair to the child node when receiving a connection request from the child node; wherein, the asymmetric key pair is pre-generated; a decryption sub-module, configured to receive a registration request sent by the child node and decrypt the symmetric encryption key and node information in the registration request by using the private key; wherein, the symmetric encryption key and node information in the registration request are pre-encrypted by the child node by using the public key; a registration sub-module, configured to perform registration on the local side according to the node information, associate the symmetric encryption key with the corresponding child node, and send a registration success notification to the child node.

[0021] The above technical solution has the following beneficial effects: By performing real-time load calculation on child nodes and intelligently allocating the number of tasks according to the load conditions, the load balancing ability and resource utilization rate are improved; continuously monitoring the load status of child nodes to adjust the number and status of child nodes, ensuring reasonable resource allocation and improving the data processing speed; continuous monitoring can quickly adjust abnormal nodes, improve the abnormal handling speed and further improve the data processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0023] Figure 1 is a flowchart of a distributed file processing method provided by an embodiment of the present invention;

[0024] Figure 2 is a flowchart of a distribution quantity calculation method provided by an embodiment of the present invention;

[0025] Figure 3 is a flowchart of a dynamic adjustment method provided by an embodiment of the present invention;

[0026] Figure 4 is a flowchart of a node connection method provided by an embodiment of the present invention;

[0027] Figure 5 is a schematic structural diagram of a distributed file processing system provided by an embodiment of the present invention;

[0028] Figure 6 is a schematic structural diagram of a task allocation module provided by an embodiment of the present invention;

[0029] Figure 7 is a schematic structural diagram of a dynamic adjustment module provided by an embodiment of the present invention;

[0030] Figure 8 It is a schematic structural diagram of the connection module provided by an embodiment of the present invention.

[0031] Reference numerals: 100 - connection module; 1001 - public key sending sub - module; 1002 - decryption sub - module; 1003 - registration sub - module; 200 - sharding module; 300 - task allocation module; 3001 - real - time load calculation sub - module; 3002 - average load calculation sub - module; 3003 - quantity calculation sub - module; 400 - dynamic adjustment module; 4001 - load calculation sub - module; 4002 - node addition sub - module; 4003 - node reduction sub - module; 500 - splicing module. Detailed implementation manners

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0033] To solve the problem of unreasonable resource allocation in the process of file compression or decompression in the prior art, an embodiment of the present invention provides a distributed file processing method. Figure 1 It is a flowchart of the distributed file processing method provided by an embodiment of the present invention, as Figure 1 shown, and the method includes:

[0034] S1. Establish a connection with the child node and periodically receive the load information sent by the child node;

[0035] The master node starts first and is ready to receive connection and service registration requests from the child nodes. After the child nodes start, they send connection and registration requests to the master node. After successful registration, the connection relationship between the master node and the child nodes is established.

[0036] The child nodes send a heartbeat to the master node at a preset interval. The heartbeat contains the load information of the child node (such as the number of tasks to be executed, CPU usage rate, memory usage rate, disk I / O, network bandwidth usage rate, etc.), for the master node to use as a basis for task allocation.

[0037] The master node periodically receives and saves the registration information and heartbeat information of the child nodes. As an optional implementation manner, when the master node does not receive the heartbeat of a certain child node for more than a predetermined time, the child node is marked as offline. The predetermined time can be set according to the actual situation, for example, 30s.

[0038] S2. When receiving a file to be processed, slice the file to be processed and assign a serial number to each sub-file;

[0039] When the master node receives a processing request (compression or decompression request) for a file, slice the received file to be processed into predetermined sizes and sequentially number each sub-file starting from 0 after slicing.

[0040] S3. Calculate the number of sub-files that can be assigned to each slave node according to the load information, and assign sub-files to the corresponding slave nodes according to the number of sub-files that can be assigned;

[0041] The load information represents the load status of the slave nodes. According to the load status of each slave node, the number of sub-files that can be assigned to each slave node can be determined, so that the slave nodes with lower load can be assigned more sub-files, and the slave nodes with higher load are assigned fewer sub-files to achieve load balancing.

[0042] After calculating the number of sub-files that a slave node can undertake, the master node assigns the sub-files to the corresponding slave nodes, and the assignment usually adopts a certain assignment rule.

[0043] S4. Regularly adjust the number of slave nodes and the assignment of sub-files according to the load information;

[0044] Regularly (such as every minute) summarize the load information of all slave nodes, and analyze the overall performance status of the current system through the load information. Intelligently adjust the number of slave nodes according to the load situation, and reassign the sub-files after the slave nodes are adjusted. For example, when the load is high, increase the number of slave nodes, and then reassign the sub-files. When the load is low, migrate the sub-files after taking offline some low-load nodes.

[0045] S5. Receive the execution results of the sub-files from the slave nodes, sort and splice the execution results of all sub-files according to the serial numbers to obtain the final execution result of the file to be processed.

[0046] After the master node makes a task assignment, establish a task status table according to the assignment situation to record the execution status of the sub-files. After each slave node receives the sub-files, process the local sub-files in parallel and perform file compression or decompression operations. After the slave node finishes the execution, send the execution result to the master node.

[0047] Every time the master node receives the execution result of a slave node, update the execution status of the corresponding sub-file in the task status table to completed according to the serial number, calculate the task progress and send a notification to the client to inform the task completion situation, where the task progress = (the number of sub-files that have been returned / the total number of sub-files) × 100%. After the master node receives the execution results of all sub-files, sort the execution results according to the serial numbers and splice them in order into a complete file to obtain the final execution result.

[0048] If a failure occurs during the execution process, that is, the master node times out without receiving the execution result of a certain child node, the master node updates the status of the corresponding sub-file in the task status table to the abnormal status, and reallocates the sub-files of the child node according to the above allocation logic.

[0049] As an alternative implementation, Figure 2 is a flowchart of the allocation quantity calculation method provided by an embodiment of the present invention. As Figure 2 shown, calculating the number of allocable sub-files for each child node according to the load information includes:

[0050] S301. Calculate the real-time load of the child node according to the load information;

[0051] S302. Calculate the average load according to the real-time load and the number of child nodes;

[0052] S303. Select the child nodes with real-time load lower than the average load, and calculate the number of allocable sub-files for the corresponding child nodes according to the real-time load and the average load.

[0053] Calculate the real-time load according to the parameters reflecting the load situation in the load information to measure the load situation of the child node at the current moment.

[0054] Calculate the average load according to the real-time loads of all child nodes. Comparing the average load with the real-time loads of each child node can determine the relative load level of the child nodes. Average load = sum of real-time loads of all child nodes / number of child nodes.

[0055] Select the child nodes with real-time load lower than the average load. The task allocation adopts proportional allocation based on the average load. The principle is that the lower the load of the child node, the more sub-files are allocated, and allocate the sub-file processing tasks for the child nodes with real-time load lower than the average load.

[0056] As an alternative implementation, calculating the real-time load of the child node according to the load information includes:

[0057] Perform weighted summation according to the CPU usage rate, memory usage rate, disk I / O usage rate, network bandwidth usage rate and task quantity in the load information to obtain the real-time load, which is calculated by the following formula:

[0058] Real-time load = CPU weight × CPU usage rate + memory weight × memory usage rate + disk I / O weight × disk I / O usage rate + network bandwidth weight × network bandwidth usage rate + task quantity weight × task quantity.

[0059] As an alternative implementation, considering that the compression / decompression task is computationally intensive, the values of each resource weight are as follows: CPU weight = 0.35; memory weight = 0.25; disk I / O weight = 0.2; network bandwidth weight = 0.1; task quantity weight = 0.1.

[0060] Calculate the number of allocable sub-files for the corresponding sub-node according to the real-time load and the average load, and calculate through the following formula:

[0061]

[0062] Where, T i represents the number of allocable sub-files corresponding to the i-th sub-node; N represents the total number of sub-files; Load i represents the real-time load of the current computing sub-node i, Load ave is the average load, Load j represents the real-time load of the j-th sub-node, and n represents the total number of sub-nodes.

[0063] It should be noted that when the calculation result is a decimal, rounding down is adopted. If there are remaining unallocated sub-files at the end, they will be uniformly allocated to the sub-node with the lowest load at the current moment.

[0064] As an alternative implementation, Figure 3 is the flowchart of the dynamic adjustment method provided by the embodiment of the present invention. As shown in Figure 3 shown, adjust the number of sub-nodes and the allocation of sub-files according to the load information at regular intervals, including:

[0065] S401. Calculate the real-time load and the average load of the sub-node according to the load information;

[0066] S402. When the average load is greater than the preset high load threshold and the duration is greater than the first preset time threshold, establish a connection with the standby node, recalculate the number of allocable sub-files for each sub-node according to the load information, and allocate sub-files to the corresponding sub-node according to the number of allocable sub-files;

[0067] S403. When the average load is lower than the preset low load threshold and the duration is greater than the second preset time threshold, identify the low-load sub-node according to the real-time load, and migrate the sub-files of the low-load sub-node to other sub-nodes.

[0068] During the processing of the sub-node, calculate the real-time load of each sub-node according to the load information of the sub-node at regular intervals, and calculate the average load according to the real-time load. The calculation method is the same as the calculation method when allocating sub-files.

[0069] Set thresholds for high load, normal load, and low load in advance. For example, set the high load threshold to 80% of the total load. When the average load exceeds 80% of the total load, it is considered a high load state; set the low load threshold to 40% of the total load. When the average load is lower than 40% of the total load, it is considered a low load state.

[0070] When in a high load state and the duration is greater than the first preset time threshold, trigger the operation of adding child nodes.

[0071] Adding child nodes includes: the master node sending a start request to the reserved standby node pool; the standby node completing initialization according to the child node start process and registering with the master node to establish a connection. After the new child node joins, the master node recalculates the real-time load and average load of each child node, and then calculates the number of distributable sub-files of each child node at the current moment and performs a reallocation, ensuring that the new child node can quickly join the actual work.

[0072] When in a low load state and the duration is greater than the second preset time threshold, consider reducing the active child nodes to save resources.

[0073] Reducing child nodes includes: the master node identifying the child node with the lowest load or the child node not assigned tasks, regarding these child nodes as nodes that can be taken offline, and migrating the tasks on these nodes to other child nodes.

[0074] As an alternative implementation Figure 4 is the flowchart of the node connection method provided by the embodiment of the present invention. As Figure 4 shown, establishing a connection with a child node includes:

[0075] S101. When receiving a connection request from a child node, send the public key in the asymmetric key pair to the child node; wherein, the asymmetric key pair is generated in advance;

[0076] S102. Receive the registration request sent by the child node, and decrypt the symmetric encryption key and node information in the registration request through the private key; wherein, the symmetric encryption key and node information in the registration request are pre-encrypted by the child node through the public key;

[0077] S103. Register on the local side according to the node information, associate the symmetric encryption key with the corresponding child node, and send a registration success notification to the child node.

[0078] After the master node is started, a pair of public and private key pairs for asymmetric encryption (such as RSA) are generated for data transmission encryption.

[0079] After the child node starts, it sends a request to the master node to obtain the master node's public key for subsequent encrypted communication. The child node generates a random symmetric encryption key (such as AES-256) and encrypts it with the master node's public key. The child node sends a service registration request to the master node, which includes node information (IP address, port number, etc.), and sends the encrypted symmetric encryption key to the master node.

[0080] After the master node receives the service registration request from the child node, it uses its own private key to decrypt the symmetric encryption key provided by the child node; the master node saves the symmetric encryption key and associates it with the child node for future use in encrypting / decrypting communication data; the master node replies with a confirmation message to the child node, informing it that the registration is successful.

[0081] An embodiment of the present invention also provides a distributed file processing system. Figure 5 It is a schematic structural diagram of the distributed file processing system provided by the embodiment of the present invention, as Figure 5 shown, the system includes:

[0082] The connection module 100 is used to establish a connection with the child node and periodically receive the load information sent by the child node.

[0083] The master node starts first and is ready to receive connections and service registration requests from the child nodes. After the child node starts, it sends connection and registration requests to the master node, and after successful registration, a connection relationship between the master node and the child node is established.

[0084] The child node sends a heartbeat to the master node at a preset interval. The heartbeat contains the load information of the child node (such as the number of tasks to be executed, CPU usage rate, memory usage rate, disk I / O, network bandwidth usage rate, etc.), for the master node to use as a basis for task allocation.

[0085] The master node periodically receives and saves the registration information and heartbeat information of the child nodes. As an optional implementation, when the master node does not receive the heartbeat of a certain child node for more than a predetermined time, that child node is marked as offline. The predetermined time can be set according to the actual situation, for example, it is 30s.

[0086] The sharding module 200 is used to shard the file to be processed and assign a sequence number to each sub-file when receiving the file to be processed.

[0087] When the master node receives a processing request (compression or decompression request) for a file, it shards the received file to be processed according to a predetermined size and sequentially numbers each sub-file after sharding starting from 0.

[0088] The task allocation module 300 is used to calculate the number of sub-files that can be allocated to each child node according to the load information, and allocate sub-files to the corresponding child nodes according to the number of sub-files that can be allocated;

[0089] The load information represents the load status of the child nodes. According to the load status of each child node, the number of sub-files that can be allocated to each child node can be determined, so that the child nodes with lower load can be allocated more sub-files, and the child nodes with higher load are allocated fewer sub-files to achieve load balancing.

[0090] After calculating the number of sub-files that a child node can undertake, the master node allocates the sub-files to the corresponding child nodes, and the allocation usually adopts a certain allocation rule.

[0091] The dynamic adjustment module 400 is used to adjust the number of child nodes and the allocation of sub-files according to the load information regularly;

[0092] Regularly (such as every minute) summarize the load information of all child nodes, and analyze the overall performance status of the current system through the load information. Intelligently adjust the number of child nodes according to the load situation, and re-allocate sub-files after the child nodes are adjusted. For example, when the load is high, increase the number of child nodes, and then re-allocate sub-files. When the load is low, migrate sub-files after taking offline some low-load nodes.

[0093] The splicing module 500 is used to receive the execution results of sub-files from child nodes, sort and splice the execution results of all sub-files according to the serial numbers to obtain the final execution result of the file to be processed.

[0094] After the master node performs task allocation, it creates a task status table according to the allocation situation to record the execution status of sub-files. After each child node receives the sub-files, it processes the local sub-files in parallel and performs file compression or decompression operations. After the child node finishes the execution, it sends the execution result to the master node.

[0095] Every time the master node receives the execution result of a child node, it updates the execution status of the corresponding sub-file in the task status table to completed according to the serial number, calculates the task progress and sends a notification to the client to inform the task completion situation, where the task progress = (the number of sub-files returned / the total number of sub-files) × 100%. After the master node receives the execution results of all sub-files, it sorts the execution results according to the serial numbers and splices them into a complete file in order to obtain the final execution result.

[0096] If an execution failure occurs during the execution process, that is, the master node times out and does not receive the execution result of a certain child node, the master node updates the status of the corresponding sub-file in the task status table to an abnormal status, and re-allocates the sub-files of the child node according to the above allocation logic.

[0097] As an optional implementation manner,Figure 6 is a schematic structural diagram of a task allocation module provided by an embodiment of the present invention. As Figure 6 shown, the task allocation module 300 includes:

[0098] A real-time load calculation sub-module 3001, configured to calculate the real-time load of a child node according to load information;

[0099] An average load calculation sub-module 3002, configured to calculate an average load according to the real-time load and the number of child nodes;

[0100] A quantity calculation sub-module 3003, configured to select a child node with a real-time load lower than the average load, and calculate the number of distributable sub-files of the corresponding child node according to the real-time load and the average load.

[0101] Perform real-time load calculation according to the parameters reflecting the load situation in the load information to measure the load situation of the child node at the current moment.

[0102] Calculate the average load according to the real-time loads of all child nodes. Comparing the average load with the real-time loads of each child node can determine the relative load level of the child nodes. Average load = sum of real-time loads of all child nodes / number of child nodes.

[0103] Select a child node with a real-time load lower than the average load. Task allocation adopts proportional allocation based on the average load. The principle is that the lower the load of a child node, the more sub-files are allocated. Allocate sub-file processing tasks to child nodes with a real-time load lower than the average load.

[0104] As an optional implementation manner, calculating the real-time load of a child node according to load information includes:

[0105] Perform weighted summation according to the CPU usage rate, memory usage rate, disk I / O usage rate, network bandwidth usage rate, and number of tasks in the load information to obtain the real-time load, which is calculated by the following formula:

[0106] Real-time load = CPU weight × CPU usage rate + memory weight × memory usage rate + disk I / O weight × disk I / O usage rate + network bandwidth weight × network bandwidth usage rate + task quantity weight × number of tasks.

[0107] As an optional implementation manner, considering that the compression / decompression task is a compute-intensive task, the weights of each resource are: CPU weight = 0.35; memory weight = 0.25; disk I / O weight = 0.2; network bandwidth weight = 0.1; task quantity weight = 0.1.

[0108] Calculate the number of distributable sub-files of the corresponding child node according to the real-time load and the average load, which is calculated by the following formula:

[0109]

[0110] Among them, T i represents the number of allocable sub-files corresponding to the i-th child node; N represents the total number of sub-files; Load i represents the real-time load of the current computing child node i, and Load ave is the average load, and Load j represents the real-time load of the j-th child node, and n represents the total number of child nodes.

[0111] It should be noted that when the calculation result is a decimal, rounding down is adopted. If there are remaining unallocated sub-files at the end, they will be uniformly allocated to the child node with the lowest load at the current moment.

[0112] As an optional implementation manner, Figure 7 is a schematic structural diagram of the dynamic adjustment module provided by the embodiment of the present invention. As Figure 7 shown, the dynamic adjustment module 400 includes:

[0113] A load calculation sub-module 4001, configured to calculate the real-time load and average load of the child nodes according to the load information;

[0114] A node addition sub-module 4002, configured to establish a connection with a standby node when the average load is greater than a preset high load threshold and the duration is greater than a first preset time threshold, re-calculate the number of allocable sub-files of each child node according to the load information, and allocate sub-files to the corresponding child nodes according to the number of allocable sub-files;

[0115] A node reduction sub-module 4003, configured to identify low-load child nodes according to the real-time load when the average load is lower than a preset low load threshold and the duration is greater than a second preset time threshold, and migrate the sub-files of the low-load child nodes to other child nodes.

[0116] During the processing of the child nodes, the real-time load of each child node is calculated regularly according to the load information of the child nodes, and the average load is calculated according to the real-time load. The calculation method is the same as the calculation method when allocating sub-files.

[0117] Thresholds for high load, normal load, and low load are preset in advance. For example, the preset high load threshold is 80% of the total load. When the average load exceeds 80% of the total load, it is considered a high load state; the preset low load threshold is 40% of the total load. When the average load is lower than 40% of the total load, it is considered a low load state.

[0118] When in a high load state and the duration is greater than the first preset time threshold, an operation to add a child node is triggered.

[0119] Adding a child node includes: the master node sending a startup request to the reserved spare node pool; the spare node completing initialization according to the child node startup process and registering with the master node to establish a connection. After the new child node joins, the master node recalculates the real-time load and average load of each child node, and then calculates the number of distributable sub-files of each child node at the current moment, and performs reallocation to ensure that the new child node can quickly join the actual work.

[0120] When in a low-load state and the duration is greater than the second preset time threshold, consider reducing the active child nodes to save resources.

[0121] Reducing a child node includes: the master node identifying the child node with the lowest load or the child node that has not been assigned tasks, taking these child nodes as the nodes that can be taken offline, and migrating the tasks on these nodes to other child nodes.

[0122] As an optional implementation manner, Figure 8 is a schematic structural diagram of the connection module provided by an embodiment of the present invention, as Figure 8 shown, the connection module 100 includes:

[0123] The public key sending sub-module 1001 is used to send the public key in the asymmetric key pair to the child node when receiving the connection request of the child node; wherein, the asymmetric key pair is pre-generated;

[0124] The decryption sub-module 1002 is used to receive the registration request sent by the child node and decrypt the symmetric encryption key and node information in the registration request through the private key; wherein, the symmetric encryption key and node information in the registration request are pre-encrypted by the child node with the public key;

[0125] The registration sub-module 1003 is used to register on its own according to the node information, associate the symmetric encryption key with the corresponding child node, and send a registration success notification to the child node.

[0126] After the master node starts, it generates a pair of public and private keys for asymmetric encryption (such as RSA) for data transmission encryption.

[0127] After the child node starts, it sends a request to the master node to obtain the public key of the master node for subsequent encrypted communication. The child node generates a random symmetric encryption key (such as AES-256), encrypts it with the public key of the master node, and the child node sends a service registration request to the master node, including node information (IP address, port number, etc.), and sends the encrypted symmetric encryption key to the master node.

[0128] After the master node receives the service registration request from the child node, it decrypts the symmetric encryption key provided by the child node using its own private key; the master node saves the symmetric encryption key and associates it with the child node for future use in encrypting / decrypting communication data; the master node replies with a confirmation message to the child node, informing it that the registration has been successful.

[0129] The above technical solution has the following beneficial effects: By performing real-time load calculation on the child nodes and intelligently allocating the number of tasks according to the load conditions, the load balancing ability and resource utilization rate are improved; continuously monitoring the load status of the child nodes to adjust the number and status of the child nodes to ensure reasonable resource allocation and improve the data processing speed; continuous monitoring can quickly adjust abnormal nodes to improve the abnormal handling speed and further improve the data processing speed.

[0130] The specific implementation manners of the above invention further elaborate on the purpose, technical solution and beneficial effects of the present invention. It should be understood that the above content is only the specific implementation manners of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A distributed file processing method, characterized in that: include: Establish a connection with the child node and receive load information sent by the child node at regular intervals; When receiving a file to be processed, the file to be processed is divided into pieces and a sequence number is assigned to each piece of sub-file; Calculate the number of allocatable sub-files of each sub-node according to the load information, and allocate sub-files to the corresponding sub-node according to the number of allocatable sub-files; Regularly adjusting the number of subnodes and the allocation of subfiles according to the load information; The execution result of the child node on the child file is received, and the execution results of all the child files are sorted and spliced ​​according to the sequence numbers to obtain the final execution result of the file to be processed.

2. The distributed file processing method according to claim 1, characterized in that: The calculating the number of allocatable sub-files of each sub-node according to the load information includes: Calculate the real-time load of the child node according to the load information; Calculate the average load according to the real-time load and the number of child nodes; A subnode whose real-time load is lower than the average load is selected, and the number of allocatable subfiles of the corresponding subnode is calculated according to the real-time load and the average load.

3. The distributed file processing method according to claim 2, characterized in that: The calculating the real-time load of the subnode according to the load information includes: The real-time load is obtained by taking a weighted sum of the CPU usage, memory usage, disk IO usage, network bandwidth usage and number of tasks in the load information. The number of allocable sub-files of the corresponding sub-node is calculated according to the real-time load and the average load, and is calculated by the following formula: Among them, T i Indicates the number of allocable sub-files corresponding to the i-th child node; N indicates the total number of sub-files; Load i Indicates the real-time load of the current computing child node i, Load ave is the average load, Load j represents the real-time load of the jth child node, and n represents the total number of child nodes.

4. The distributed file processing method according to claim 1, characterized in that: The timing adjusting the number of subnodes and the allocation of subfiles according to the load information includes: Calculate the real-time load and average load of the subnode according to the load information; When the average load is greater than the preset high load threshold and the duration is greater than the first preset time threshold, a connection is established with the standby node, the number of allocatable subfiles of each subnode is recalculated according to the load information, and subfiles are allocated to the corresponding subnode according to the number of allocatable subfiles; When the average load is lower than a preset low load threshold and the duration is greater than a second preset time threshold, a low load subnode is identified according to the real-time load, and the subfiles of the low load subnode are migrated to other subnodes.

5. The distributed file processing method according to claim 1, characterized in that: The establishing of a connection with a child node comprises: When receiving a connection request from a child node, sending a public key in an asymmetric key pair to the child node; wherein the asymmetric key pair is generated in advance; Receive a registration request sent by the child node, and decrypt the symmetric encryption key and node information in the registration request by using the private key; wherein the symmetric encryption key and node information in the registration request are pre-encrypted by the child node by using the public key; Register on the local side according to the node information, associate the symmetric encryption key with the corresponding child node, and send a registration success notification to the child node.

6. A distributed file processing system, characterized in that: include: A connection module is used to establish a connection with a subnode and periodically receive load information sent by the subnode; A fragmentation module is used to fragment the file to be processed when it is received, and assign a sequence number to each sub-file; A task allocation module, used for calculating the number of allocatable sub-files of each sub-node according to the load information, and allocating sub-files to the corresponding sub-node according to the number of allocatable sub-files; A dynamic adjustment module, used to adjust the number of sub-nodes and the allocation of sub-files according to the load information at regular intervals; The splicing module is used to receive the execution results of the child nodes on the child files, sort the execution results of all the child files according to the sequence numbers, and then splice them to obtain the final execution result of the file to be processed.

7. The distributed file processing system according to claim 6, characterized in that: The task allocation module includes: A real-time load calculation submodule, used to calculate the real-time load of the subnode according to the load information; An average load calculation submodule, used to calculate the average load according to the real-time load and the number of subnodes; The quantity calculation submodule is used to select a subnode whose real-time load is lower than the average load, and calculate the number of allocable subfiles of the corresponding subnode according to the real-time load and the average load.

8. The distributed file processing system according to claim 7, characterized in that: The calculating the real-time load of the subnode according to the load information includes: The real-time load is obtained by taking a weighted sum of the CPU usage, memory usage, disk IO usage, network bandwidth usage and number of tasks in the load information. The number of allocable sub-files of the corresponding sub-node is calculated according to the real-time load and the average load, and is calculated by the following formula: Among them, T i Indicates the number of allocable sub-files corresponding to the i-th child node; N indicates the total number of sub-files; Load i Indicates the real-time load of the current computing child node i, Load ave is the average load, Load j represents the real-time load of the jth child node, and n represents the total number of child nodes.

9. The distributed file processing system according to claim 6, characterized in that: The dynamic adjustment module comprises: A load calculation submodule, used to calculate the real-time load and average load of the subnode according to the load information; A node adding submodule is used to establish a connection with a standby node when the average load is greater than a preset high load threshold and the duration is greater than a first preset time threshold, recalculate the number of allocatable subfiles of each subnode according to the load information, and allocate subfiles to the corresponding subnode according to the number of allocatable subfiles; The node reduction submodule is used to identify low-load subnodes according to real-time load and migrate subfiles of the low-load subnodes to other subnodes when the average load is lower than a preset low-load threshold and the duration is greater than a second preset time threshold.

10. The distributed file processing system according to claim 6, characterized in that: The connection module comprises: A public key sending submodule, used for sending the public key in the asymmetric key pair to the child node when receiving a connection request from the child node; wherein the asymmetric key pair is generated in advance; A decryption submodule, used to receive a registration request sent by a child node, and decrypt the symmetric encryption key and node information in the registration request by using a private key; wherein the symmetric encryption key and node information in the registration request are pre-encrypted by the child node by using the public key; The registration submodule is used to register on the local side according to the node information, associate the symmetric encryption key with the corresponding child node, and send a registration success notification to the child node.

Images