Parallel data processing method and device, computer device and storage medium

Abstract

The application discloses a parallel data processing method and device, computer equipment and a storage medium, comprising: selecting a master node and at least two slave nodes in a plurality of preset computing nodes through a preset promotion protocol; the master node analyzes target data to be processed, splits the target data into parallel tasks with the same number as the slave nodes, and distributes the parallel tasks to the at least two slave nodes; the at least two slave nodes each perform data operation on the parallel task belonging to the slave node, each generate a processing result corresponding to the parallel task, and upload the processing result to the master node; the master node receives the processing result uploaded by the at least two slave nodes respectively, combines the processing result, and sends the combined processing result to a preset target terminal, wherein the target data is uploaded by the target terminal. Through the master node, unified task allocation of the slave nodes is realized, and efficient and automatic processing of parallel data is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a parallel data processing method, apparatus, computer device and storage medium. Background Technology

[0002] Parallel processing is a computational method in computer systems that enables the simultaneous execution of two or more processes. Parallel processing can work on different aspects of the same program at the same time. The main purpose of parallel processing is to save time in solving large and complex problems. To use parallel processing, the program must first be parallelized, that is, the different parts of the work are distributed among different processing processes (threads).

[0003] The inventors of this invention discovered in their research that in the prior art, in data processing involving multiple computer nodes, parallel processing suffers from the problem of inter-device correlation, making it difficult to achieve automated parallel processing. Summary of the Invention

[0004] This invention provides a parallel data processing method, apparatus, computer device, and storage medium that enables unified management across distributed computing nodes.

[0005] To solve the above-mentioned technical problems, one technical solution adopted by the embodiments of the present invention is: to provide a parallel data processing method, including:

[0006] A master node and at least two slave nodes are elected from a set of predefined computing nodes using a predefined election protocol.

[0007] The master node parses the target data to be processed, splits the target data into parallel tasks equal to the number of slave nodes, and distributes the parallel tasks to the at least two slave nodes;

[0008] Each of the at least two slave nodes performs data operations on its respective parallel task, generates a processing result corresponding to the parallel task, and uploads it to the master node.

[0009] The master node receives the processing results uploaded by each of the at least two slave nodes, merges the processing results, and sends them to a preset target terminal, wherein the target data is uploaded by the target terminal.

[0010] Optionally, the step of electing a master node and at least two slave nodes from a preset set of computing nodes through a preset election protocol includes:

[0011] Collect the first operating parameters of each computing node in the previous data processing cycle;

[0012] The first operating parameters are sequentially input into a preset device scoring model, wherein the device scoring model is a neural network model pre-trained to a convergent state for scoring computer nodes;

[0013] Read the computational scores of each computing node output by the device scoring model;

[0014] The computing node with the highest computational score is selected as the master node, and the other computing nodes are designated as slave nodes.

[0015] Optionally, the step of electing a master node and at least two slave nodes from a preset set of computing nodes through a preset election protocol includes:

[0016] Collect the second operating parameters of each computing node in the previous data processing cycle;

[0017] The data of each of the second operating parameters is merged to generate merged data, and the merged data is hashed to generate a hash value;

[0018] The master node and at least two slave nodes are determined in each of the settlement nodes based on the hash value.

[0019] Optionally, determining the master node and at least two slave nodes in each of the settlement nodes based on the hash value includes:

[0020] Read the number of nodes in the computing node;

[0021] Compare the number of nodes with the hash value;

[0022] When the hash value is less than or equal to the number of nodes, the computing node represented by the node number corresponding to the hash value is determined as the master node, and the other computing nodes are determined as slave nodes.

[0023] When the hash value is greater than the number of nodes, a remainder value is generated by performing a modulo operation on the hash value based on the number of nodes. The computing node represented by the node number corresponding to the remainder value is determined as the master node, and the other computing nodes are determined as slave nodes.

[0024] Optionally, the step of electing a master node and at least two slave nodes from a preset set of computing nodes through a preset election protocol includes:

[0025] Within the preset countdown interval, each computing node randomly generates a countdown task;

[0026] Once the countdown task is completed, each computing node sends an election request to other computing nodes and receives an election response from the other computing nodes.

[0027] When any computing node receives more than half the number of election responses, that computing node is determined to be the master node, and the remaining computing nodes are determined to be slave nodes.

[0028] Optionally, the master node parses the target data to be processed, splits the target data into parallel tasks equal to the number of slave nodes, and distributes the parallel tasks to the at least two slave nodes, including:

[0029] The master node performs data slicing on the target data to generate the parallel tasks, and stores the parallel tasks sequentially in the task queue;

[0030] The master node generates a first task thread and distributes the parallel tasks in the task queue to the at least two slave nodes through the first task thread.

[0031] Optionally, each of the at least two subordinate nodes performs data computation on its respective parallel task, and each generates a processing result corresponding to the parallel task, including:

[0032] The main thread of each of the at least two slave nodes starts a task queue and creates a first child thread to continuously receive parallel tasks from the main node.

[0033] The main thread creates a custom task pool to process the parallel tasks and distributes the parallel tasks to a second child thread that has been initialized and started JTA transactions.

[0034] The main thread receives the processing result of the parallel task from the second sub-thread and sends the processing result to the main node.

[0035] To address the aforementioned technical problems, embodiments of the present invention also provide a parallel data processing apparatus, comprising:

[0036] The election module is used to elect a master node and at least two slave nodes from a set of preset computing nodes through a preset election protocol.

[0037] The processing module is used by the master node to parse the target data to be processed, split the target data into parallel tasks with the same number of slave nodes, and distribute the parallel tasks to the at least two slave nodes;

[0038] The computation module is used for each of the at least two slave nodes to perform data computation on their respective parallel tasks, generate processing results corresponding to the parallel tasks, and upload them to the master node.

[0039] An execution module is used for the master node to receive the processing results uploaded by each of the at least two slave nodes, and to merge the processing results and send them to a preset target terminal, wherein the target data is uploaded by the target terminal.

[0040] Optionally, the parallel data processing device further includes:

[0041] The first acquisition submodule is used to acquire the first operating parameters of each computing node in the previous data processing cycle;

[0042] The first classification submodule is used to input the first operating parameters sequentially into a preset device scoring model, wherein the device scoring model is a neural network model pre-trained to a convergent state for scoring computer nodes;

[0043] The first reading submodule is used to read the calculation scores of each calculation node output by the device scoring model;

[0044] The first confirmation submodule is used to select the computing node with the highest calculation score as the master node and determine the other computing nodes as slave nodes.

[0045] Optionally, the parallel data processing device further includes:

[0046] The second acquisition submodule is used to acquire the second operating parameters of each computing node in the previous data processing cycle;

[0047] The first merging submodule is used to merge the data of each of the second operating parameters to generate merged data, and to perform a hash operation on the merged data to generate a hash value;

[0048] The second confirmation submodule is used to determine the master node and at least two slave nodes in each of the settlement nodes based on the hash value.

[0049] Optionally, the parallel data processing device further includes:

[0050] The first reading submodule is used to read the number of nodes of the computing node;

[0051] The first comparison submodule is used to compare the number of nodes with the hash value;

[0052] The first execution submodule is used to determine the computing node represented by the node number corresponding to the hash value as the master node and the other computing nodes as slave nodes when the hash value is less than or equal to the number of nodes.

[0053] The second execution submodule is used to perform a modulo operation on the hash value according to the number of nodes when the hash value is greater than the number of nodes to generate a remainder value, and to determine the computing node represented by the node number corresponding to the remainder value as the master node, and to determine the other computing nodes as slave nodes.

[0054] Optionally, the parallel data processing device further includes:

[0055] The first generation submodule is used to randomly generate a countdown task for each computing node within a preset countdown interval.

[0056] The first processing submodule is used to send an election request to other computing nodes and receive the election response from the other computing nodes after the countdown task is completed.

[0057] The third execution submodule is used to determine the computing node as the master node and the remaining computing nodes as slave nodes when any computing node receives more than half of the number of election responses.

[0058] Optionally, the parallel data processing device further includes:

[0059] The second processing submodule is used by the master node to perform data slicing on the target data to generate the parallel tasks, and to store the parallel tasks in the task queue in sequence;

[0060] The fourth execution submodule is used by the master node to generate a first task thread and distribute the parallel tasks in the task queue to the at least two slave nodes through the first task thread.

[0061] Optionally, the parallel data processing device further includes:

[0062] The third processing submodule is used to start the task queue of each of the at least two slave nodes' main threads and create a first subthread to continuously receive parallel tasks from the main nodes.

[0063] The fourth processing submodule is used by the main thread to create a custom task pool for processing the parallel tasks and to distribute the parallel tasks to the second sub-thread that has been initialized and started JTA transactions.

[0064] The fifth execution submodule is used by the main thread to receive the processing result of the parallel task by the second sub-thread and send the processing result to the main node.

[0065] To address the aforementioned technical problems, this invention also provides a computer device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the parallel data processing method described above.

[0066] To address the aforementioned technical problems, this embodiment of the invention also provides a computer storage medium, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors perform the steps of the parallel data processing method described above.

[0067] The beneficial effects of this invention are as follows: In a multi-computing node system, a master node and slave nodes are selected from multiple computing nodes through a selection process. The master node receives the target data, performs unified scheduling and task splitting on the target data, and then distributes the split parallel data to each slave node. The slave nodes perform calculations on the parallel tasks and obtain multiple processing results. The master node merges the multiple processing results to generate the processing result for the target node and then sends the total processing result to the corresponding target terminal. This allows for unified task allocation between the master node and slave nodes, achieving efficient and automated processing of parallel data. Attached Figure Description

[0068] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0069] Figure 1 This is a basic flowchart illustrating a parallel data processing method according to a specific embodiment of this application;

[0070] Figure 2 A schematic diagram illustrating a first implementation of the recommendation protocol according to a specific embodiment of this application;

[0071] Figure 3 This is a schematic diagram illustrating a second implementation of the recommendation protocol according to a specific embodiment of this application;

[0072] Figure 4 This is a schematic diagram illustrating the process of determining the master node based on the hash value according to a specific embodiment of this application;

[0073] Figure 5 This is a schematic diagram illustrating a third implementation of the recommendation protocol according to a specific embodiment of this application;

[0074] Figure 6 This is a schematic diagram of the parallel task distribution process according to a specific embodiment of this application;

[0075] Figure 7 A flowchart illustrating the parallel task processing of a parameter node according to one embodiment of this application;

[0076] Figure 8 This is a schematic diagram of the basic structure of a parallel data processing device according to an embodiment of this application;

[0077] Figure 9 This is a basic structural block diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0078] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0079] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the word “comprising” as used in the specification of this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0080] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0081] Those skilled in the art will understand that the term "terminal" as used herein includes both devices that are wireless signal receivers, devices that are wireless signal receivers without transmitting capability, and devices with receiving and transmitting hardware, having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such devices may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptop and / or handheld computers or other devices that have and / or include a radio frequency receiver. As used herein, "terminal" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally, and / or in a distributed manner, operating in any other location on Earth and / or in space. "Terminal" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0082] Please see Figure 1 , Figure 1 This is a schematic diagram of the basic flow of the parallel data processing method in this embodiment. Figure 1 As shown, a parallel data processing method includes:

[0083] S110. Elect a master node and at least two slave nodes from a set of preset computing nodes using a preset election protocol.

[0084] The parallel data processing method in this embodiment is applicable to distributed systems with multiple computing nodes. Multiple computing nodes have the same function: performing data operations. When performing data operations, a master node needs to be elected from among the multiple computing nodes. The master node is responsible for scheduling tasks related to the target data to be processed.

[0085] The generation of the master node is not fixed but dynamic. Within a data processing cycle, a master node needs to be elected. Once the master node is generated, the other computing nodes automatically become slave nodes. Slave nodes receive tasks assigned by the master node and complete the computation of the assigned tasks.

[0086] The master node is generated through a recommendation protocol. The first recommendation method involves using a neural network model to calculate the running status of each computing node and selecting the node with the best running status as the master node. This method is suitable for applications requiring a master node with powerful computing capabilities.

[0087] Specifically, the first operating parameters of each computing node in the previous data processing cycle are collected. The first operating parameters are used to characterize the operating status of each computing node in the previous data processing cycle. Therefore, the first operating parameters include (but are not limited to): the data processing speed, data transmission time, accuracy of data processing results, memory utilization, or other parameters that can characterize the computing efficiency of each computing node.

[0088] The collected initial operating parameters are sequentially input into a pre-set device scoring model. This model is a neural network model pre-trained to convergence, used to score computer nodes. The device scoring model consists of one or a variant of a convolutional neural network (CNN), a deep convolutional neural network (DCNN), or a recurrent neural network (RNN). After initialization, the initialized model undergoes supervised training using labeled data until convergence, resulting in the final model. The converged device scoring model can extract and classify features from the initial operating parameters to obtain the computational scores for each computing node.

[0089] After determining the computation score of each computation node, the computation node with the highest computation score is selected as the master node. After the master node is determined, the master node sends a master node confirmation message to each other node. After receiving the message, the other nodes automatically confirm themselves as slave nodes and wait for the task data from the master node.

[0090] The second recommendation method of the recommendation protocol is as follows: Second running parameters of each computing node in the previous data processing cycle are collected, where the first and second running parameters are of the same category. Then, the second running parameters of each computing node are merged to generate merged data. The hash value of the merged data is calculated using a hash function. After obtaining the hash value, the master node is selected from among the numerous computing nodes based on the hash value. This recommendation method is designed for application scenarios with higher data security requirements.

[0091] Specifically, the second operating parameters of each computing node in the previous data processing cycle are collected. The second operating parameters are used to characterize the operating status of each computing node in the previous data processing cycle. Therefore, the second operating parameters include (but are not limited to): the data processing speed of each computing node, the data transmission time, the accuracy of the data processing results, the memory utilization rate, or other parameters that can characterize the computing efficiency.

[0092] After obtaining the second operating parameters of each computing node, the second operating parameters of all nodes are merged into a data table. Then, a hash operation is performed on all the second operating parameters in the data table using a preset hash calculation formula. The result of the hash operation is a hash value used to represent the second operating parameters of all nodes.

[0093] After obtaining the hash value, this hash value may be less than or equal to the number of computing nodes in the system, or it may be greater than the number of computing nodes in the system. Different methods are used to determine the master node for these two different situations. First, each node in the system has a node number from 1 to n (n >= 3), where n equals the number of nodes. When the hash value is less than or equal to the number of computing nodes in the system, the computing node with the node number corresponding to the hash value is directly determined as the master node. When the hash value is greater than the number of computing nodes in the system, n is used as the divisor and a modulo operation is performed on the hash value. The remainder obtained is less than or equal to n, and the computing node with the node number corresponding to the remainder is then determined as the master node. After the master node is determined, it sends a master node confirmation message to all other nodes. Upon receiving this message, other nodes automatically confirm themselves as slave nodes and wait for the master node's task data. The master node elected in this way is entirely determined by the second operating parameters of all nodes within the aforementioned data processing cycle, resulting in strong unpredictability and preventing targeted attacks on the master node in the system.

[0094] The third election method in the election protocol is determined through election requests among computing nodes. When more than half of the computing nodes elect the same node as the master node, that node becomes the master node, and other nodes are automatically confirmed as slave nodes after the master node's confirmation. This election method is suitable for application scenarios with higher data security requirements. It also effectively prevents faulty nodes from being elected as master nodes, ensuring system stability.

[0095] Specifically, when each data processing cycle of the upper layer is completed, or when the elected master node is in a faulty state, each computing node, based on its own operating status and after the previous task has been executed, randomly generates a countdown task within a preset countdown interval (150ms-300ms) according to a countdown script based on a threshold. Once the countdown time is generated, each computing node generates a countdown task based on the countdown time.

[0096] Once each node's countdown task is completed (i.e., the countdown timer reaches zero), the computing node sends an election request to all other computing nodes. Upon receiving the election request, if other computing nodes have not yet completed their own countdown tasks, they reply to the request agreeing to elect the sending node as the master node and send an election response to that node. If other computing nodes have already completed their countdown tasks when they receive the election request, they do not reply, meaning they do not agree to the sending node becoming the master node, but instead send their own election requests to other nodes, requesting to be elected as the master node. In this way, the computing node that first sends the election request has the highest probability of receiving a response and is the first to receive more than half the votes, thus being successfully elected as the new master node. After the master node is determined, it sends a master node confirmation message to all other nodes. Upon receiving this message, other nodes automatically confirm themselves as slave nodes and await task data from the master node. Because faulty nodes cannot send or receive election information, this election method effectively avoids faulty nodes participating in computation, improving system smoothness and stability. The randomness of the scheduled tasks also prevents the master node from being known and attacked by others, increasing security.

[0097] In this embodiment, there are two subordinate nodes. However, depending on the specific application scenario, in some embodiments, the number of subordinate nodes can be three, four, five or more.

[0098] S120. The master node parses the target data to be processed, splits the target data into parallel tasks with the same number of slave nodes, and distributes the parallel tasks to the at least two slave nodes.

[0099] Once the master node is determined, it receives the target data sent by the target node. After receiving the target data, the master node slices the target data according to the number of slave nodes, generating parallel tasks with the same number of slave nodes.

[0100] The master node stores individual parallel tasks in a queue. Then, it creates two threads: one thread distributes the tasks in the queue to all slave nodes at once, and the other thread receives the processing results uploaded by each slave node, calculates the exception rate, merges the processing results, and synchronously records the data processing process to the master node's log.

[0101] S130. Each of the at least two subordinate nodes performs data operations on its respective parallel task, generates a processing result corresponding to the parallel task, and uploads it to the master node.

[0102] After each slave node receives a parallel task from the master node, the slave node's main thread starts a task queue and creates a child thread to continuously receive parallel tasks from the master node. Then, the main thread creates a custom task pool to handle the data entry tasks. A new child thread is created and initialized, and JTA transactions are started. The main thread assigns parallel tasks to the child thread, receives the processing results uploaded by the child thread, and uploads the processing results to the master node's receiving thread.

[0103] In some implementations, after receiving feedback from the child thread, the slave node's main thread retrieves the queued tasks and continues to allocate them; if the tasks have been allocated, it detects that the master node has issued a commit or rollback instruction; upon receiving the master node's transaction command, the main thread notifies all threads to perform a commit or rollback operation.

[0104] S140. The master node receives the processing results uploaded by each of the at least two slave nodes, merges the processing results, and sends them to a preset target terminal, wherein the target data is uploaded by the target terminal.

[0105] After receiving the processing results uploaded by each slave node, the master node combines the processing results of different slave nodes according to the slicing order of each parallel task during task slicing to generate the total processing result of the target data. Then, the total processing result is sent to the target terminal to complete the data processing request to the target terminal.

[0106] In the above implementation, in a multi-computing-node system, a master node and slave nodes are selected from multiple computing nodes through an election process. The master node receives the target data, performs unified scheduling and task splitting on the target data, and then distributes the split parallel data to each slave node. The slave nodes perform calculations on the parallel tasks and obtain multiple processing results. The master node merges these multiple processing results to generate the processing result for the target node and then sends the overall processing result to the corresponding target terminal. This allows for unified task allocation between the master node and slave nodes, achieving efficient and automated processing of parallel data.

[0107] In some implementations, the master node is elected using a neural network model. See [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of the first implementation of the recommendation protocol in this embodiment.

[0108] like Figure 2 As shown, S110 includes:

[0109] S111A: Collect the first operating parameters of each computing node in the previous data processing cycle;

[0110] Collect the first operating parameters of each computing node in the previous data processing cycle. The first operating parameters are used to characterize the operating status of each computing node in the previous data processing cycle. Therefore, the first operating parameters include (but are not limited to): the data processing speed, data transmission time, accuracy of data processing results, memory utilization, or other parameters that can characterize the computing efficiency of each computing node.

[0111] S112A. The first operating parameters are sequentially input into a preset device scoring model, wherein the device scoring model is a neural network model pre-trained to a convergent state for scoring computer nodes.

[0112] The collected initial operating parameters are sequentially input into a pre-set device scoring model. The device scoring model is a neural network model pre-trained to convergence, used to score computer nodes. The device scoring model consists of one or any variant of a convolutional neural network model, a deep convolutional neural network model, or a recurrent neural network model. After initialization, the initialized model is trained under supervised supervision using labeled data until convergence, resulting in the final model.

[0113] S113A: Read the calculation scores of each calculation node output by the scoring model of the device;

[0114] The device scoring model trained to convergence can extract features and classify the first operating parameters to obtain the computational scores of each computing node.

[0115] S114A. Select the computing node with the highest computational score as the master node, and determine the other computing nodes as slave nodes.

[0116] After determining the computational scores of each computing node, the computing node with the highest score is selected as the master node. Once the master node is determined, it sends a master node confirmation message to all other nodes. Upon receiving this message, the other nodes automatically confirm themselves as slave nodes and await task data from the master node. This recommendation method is designed for application scenarios requiring master nodes with powerful computing capabilities. The master node selected through a neural network model has significantly stronger computing power compared to other nodes, which helps improve the overall management efficiency of the system.

[0117] In some implementations, the master node needs to be determined using a hash value. See [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of a second implementation of the recommendation protocol in this embodiment.

[0118] like Figure 3 As shown, S110 includes:

[0119] S111B: Collect the second operating parameters of each computing node in the previous data processing cycle;

[0120] The second operating parameters of each computing node in the previous data processing cycle are collected. The second operating parameters are used to characterize the operating status of each computing node in the previous data processing cycle. Therefore, the second operating parameters include (but are not limited to): the data processing speed of each computing node, the data transmission time, the accuracy of the data processing results, the memory utilization rate, or other parameters that can characterize the computing efficiency.

[0121] S112B: Merge the data of each of the second operating parameters to generate merged data, and perform a hash operation on the merged data to generate a hash value;

[0122] After obtaining the second operating parameters of each computing node, the second operating parameters of all nodes are merged into a data table. Then, a hash operation is performed on all the second operating parameters in the data table using a preset hash calculation formula. The result of the hash operation is a hash value used to represent the second operating parameters of all nodes.

[0123] S113B. Determine the master node and at least two slave nodes in each of the settlement nodes based on the hash value.

[0124] After obtaining the hash value, this hash value may be less than or equal to the number of computing nodes in the system, or it may be greater than the number of computing nodes in the system. Different methods are used to determine the master node for these two different cases. Please refer to [link to details]. Figure 4 , Figure 4 This is a schematic diagram illustrating the process of determining the master node based on the hash value in this embodiment.

[0125] like Figure 4As shown, S113B includes:

[0126] S114B, Read the number of nodes in the computing node;

[0127] Before determining the target node based on the hash value, it is necessary to read the number of computing nodes, which is the total number of computing nodes in the system.

[0128] S115B. Compare the number of nodes with the hash value;

[0129] After obtaining the hash value, this hash value may be less than or equal to the number of computing nodes in the system, or it may be greater than the number of computing nodes in the system. Therefore, it is necessary to compare the number of nodes with the hash value.

[0130] S116B. When the hash value is less than or equal to the number of nodes, the computing node represented by the node number corresponding to the hash value is determined as the master node, and the other computing nodes are determined as slave nodes.

[0131] Each node in the system has a node number from 1 to n (n >= 3), where n is equal to the number of nodes. When the hash value is less than or equal to the number of nodes in the system, the node whose hash value corresponds to the node number is directly determined as the master node.

[0132] S117B. When the hash value is greater than the number of nodes, perform a modulo operation on the hash value according to the number of nodes to generate a remainder value. Determine the computing node represented by the node number corresponding to the remainder value as the master node, and determine the other computing nodes as slave nodes.

[0133] When the hash value exceeds the number of computing nodes in the system, n is used as the dividend and a modulo operation is performed on the hash value. If the remainder is less than or equal to n, the computing node with the node number corresponding to the remainder is determined as the master node. After the master node is determined, it sends a master node confirmation message to all other nodes. Upon receiving this message, other nodes automatically confirm themselves as slave nodes and await task data from the master node. The master node elected in this way is entirely determined by the second operating parameters of all nodes within the aforementioned data processing cycle, resulting in strong unpredictability and preventing targeted attacks on the master node in the system.

[0134] In some implementations, the master node is determined through an election request among compute nodes. See also... Figure 5 , Figure 5 This is a schematic diagram of a third implementation of the recommendation protocol in this embodiment.

[0135] like Figure 5 As shown, S110 includes:

[0136] S111C. Within the preset countdown interval, each computing node randomly generates a countdown task.

[0137] When each data processing cycle of the upper layer is completed, or when the elected master node is in a faulty state, each computing node, based on its own operating status and after the previous task has been executed, randomly generates a countdown task within a preset countdown interval (150ms-300ms) according to a threshold countdown script. Once the countdown timer is generated, each computing node generates a countdown task based on that timer.

[0138] S112C. When the countdown task is completed, each computing node sends an election request to other computing nodes and receives the election response from the other computing nodes.

[0139] Once each node's countdown task is completed (i.e., the countdown time reaches zero), the compute node sends an election request to all other compute nodes. Upon receiving the election request, if other compute nodes have not yet completed their own countdown tasks, they reply to the election request agreeing to elect the node that sent the request as the master node, and send an election reply to that node. If other compute nodes have already completed their own countdown tasks when they receive the election request, they do not reply to the election request, i.e., they do not agree to elect the node that sent the election request as the master node, but instead send their own election requests to other nodes, requesting to be elected as the master node.

[0140] S113C. When any computing node receives more than half the number of election responses, that computing node is determined to be the master node, and the remaining computing nodes are determined to be slave nodes.

[0141] Thus, the computing node that first sends the election request has the highest probability of receiving a response and is the first to receive more than half the votes, successfully being elected as the new master node. After the master node is determined, it sends a master node confirmation message to all other nodes. Upon receiving this message, other nodes automatically confirm themselves as slave nodes and await task data from the master node. Because faulty nodes cannot send or receive election information, this election method effectively avoids faulty nodes participating in computation, improving system smoothness and stability. The randomness of the scheduled tasks also prevents the master node from being known and attacked by others, increasing security.

[0142] Please see Figure 6 , Figure 6 This is a schematic diagram of the parallel task distribution process in this embodiment.

[0143] like Figure 6 The S120 shown includes:

[0144] S121. The master node performs data slicing on the target data to generate the parallel tasks, and stores the parallel tasks sequentially in the task queue;

[0145] Once the master node is determined, it receives the target data sent by the target node. After receiving the target data, the master node slices the target data according to the number of slave nodes, generating parallel tasks with the same number of slave nodes.

[0146] S122, the master node generates a first task thread, and distributes the parallel tasks in the task queue to the at least two slave nodes through the first task thread.

[0147] The master node stores individual parallel tasks in a queue, and then creates two threads. One thread is the first task thread. The first task thread is used to distribute the tasks in the queue to all slave nodes at once, while the other thread receives the processing results uploaded by each slave node, calculates the exception rate, merges the processing results, and synchronously records the data processing process to the master node's log.

[0148] Please see Figure 7 , Figure 7 This is a flowchart illustrating the process of parameter nodes handling parallel tasks in this embodiment.

[0149] like Figure 7 As shown, S130 includes:

[0150] S131. The main thread of each of the at least two slave nodes starts a task queue and creates a first child thread to continuously receive parallel tasks from the main node.

[0151] After receiving a parallel task from the master node, each slave node's main thread starts a task queue and creates a first child thread to continuously receive parallel tasks from the master node.

[0152] S132. The main thread creates a custom task pool for processing the parallel tasks and distributes the parallel tasks to a second child thread that has been initialized and started JTA transactions.

[0153] A custom task pool is created by the main thread to handle database insertion tasks; a second child thread is created and initialized, and JTA transactions are started. JTA allows applications to perform distributed transaction processing—accessing and updating data on two or more networked computer resources. JTA support in JDBC drivers greatly enhances data access capabilities.

[0154] S133. The main thread receives the processing result of the parallel task from the second sub-thread and sends the processing result to the main node.

[0155] The main thread assigns parallel tasks to child threads, receives the processing results uploaded by the child threads, and uploads the processing results to the thread on the main node for receiving.

[0156] In some implementations, after receiving feedback from the child thread, the slave node's main thread retrieves the queued tasks and continues to allocate them; if the tasks have been allocated, it detects that the master node has issued a commit or rollback instruction; upon receiving the master node's transaction command, the main thread notifies all threads to perform a commit or rollback operation.

[0157] To address the aforementioned technical problems, embodiments of the present invention also provide a parallel data processing apparatus. Please refer to [link / reference needed] for details. Figure 8 , Figure 8 This is a schematic diagram of the basic structure of the parallel data processing device in this embodiment.

[0158] like Figure 8 As shown, a parallel data processing device includes: a recommendation module 110, a processing module 120, a computation module 130, and an execution module 140. The recommendation module 110 is used to recommend one master node and at least two slave nodes from a preset set of computing nodes using a preset recommendation protocol. The processing module 120 is used by the master node to parse the target data to be processed, split the target data into parallel tasks equal to the number of slave nodes, and distribute the parallel tasks to the at least two slave nodes. The computation module 130 is used by the at least two slave nodes to perform data computation on their respective parallel tasks, generate processing results corresponding to the parallel tasks, and upload them to the master node. The execution module 140 is used by the master node to receive the processing results uploaded by the at least two slave nodes, merge the processing results, and send them to a preset target terminal, wherein the target data is uploaded by the target terminal.

[0159] In a multi-computing-node system, the parallel data processing device selects master and slave nodes from multiple computing nodes through an election process. The master node receives the target data, performs unified scheduling and task splitting, and then distributes the split parallel data to the respective slave nodes. The slave nodes perform parallel tasks and obtain multiple processing results. The master node merges these results to generate the final processing result for the target node and sends the overall result to the corresponding target terminal. This system enables efficient and automated processing of parallel data through unified task allocation from the master node to the slave nodes.

[0160] In some embodiments, the parallel data processing device further includes: a first acquisition submodule, a first classification submodule, and a first confirmation submodule. The first acquisition submodule is used to acquire the first operating parameters of each computing node in the previous data processing cycle; the first classification submodule is used to sequentially input the first operating parameters into a preset device scoring model, wherein the device scoring model is a neural network model pre-trained to convergence for scoring computer nodes; the first reading submodule is used to read the computational scores of each computing node output by the device scoring model; and the first confirmation submodule is used to select the computing node with the highest computational score as the master node and determine the other computing nodes as slave nodes.

[0161] In some embodiments, the parallel data processing device further includes: a second acquisition submodule, a first merging submodule, and a second confirmation submodule. The second acquisition submodule is used to acquire the second operating parameters of each computing node in the previous data processing cycle; the first merging submodule is used to merge the data of each of the second operating parameters to generate merged data, and to perform a hash operation on the merged data to generate a hash value; the second confirmation submodule is used to determine the master node and at least two slave nodes in each settlement node based on the hash value.

[0162] In some embodiments, the parallel data processing device further includes: a first reading submodule, a first comparison submodule, a first execution submodule, and a second execution submodule. The first reading submodule is used to read the number of computing nodes; the first comparison submodule is used to compare the number of nodes with the hash value; the first execution submodule is used to, when the hash value is less than or equal to the number of nodes, determine the computing node represented by the node number corresponding to the hash value as the master node and the other computing nodes as slave nodes; the second execution submodule is used to, when the hash value is greater than the number of nodes, perform a modulo operation on the hash value based on the number of nodes to generate a remainder value, determine the computing node represented by the node number corresponding to the remainder value as the master node, and the other computing nodes as slave nodes.

[0163] In some embodiments, the parallel data processing device further includes: a first generation submodule, a first processing submodule, and a third execution submodule. The first generation submodule is used to randomly generate a countdown task for each computing node within a preset countdown interval; the first processing submodule is used to send election requests to other computing nodes and receive election responses from the other computing nodes after the countdown task is completed; the third execution submodule is used to determine that computing node is the master node and the remaining computing nodes are slave nodes when the number of election responses received by any computing node is greater than half the number of nodes.

[0164] In some embodiments, the parallel data processing device further includes a second processing submodule and a fourth execution submodule. The second processing submodule is used by the master node to perform data slicing on the target data to generate the parallel tasks, and to sequentially store the parallel tasks in a task queue. The fourth execution submodule is used by the master node to generate a first task thread, and to distribute the parallel tasks in the task queue to the at least two slave nodes through the first task thread.

[0165] In some embodiments, the parallel data processing device further includes: a third processing submodule, a fourth processing submodule, and a fifth execution submodule. The third processing submodule is used for the main threads of the at least two slave nodes to start task queues and create first child threads to continuously receive parallel tasks from the master nodes; the fourth processing submodule is used for the main threads to create a custom task pool for processing the parallel tasks and to distribute the parallel tasks to second child threads that have been initialized and have started JTA transactions; the fifth execution submodule is used for the main threads to receive the processing results of the parallel tasks from the second child threads and send the processing results to the master node.

[0166] To address the aforementioned technical problems, embodiments of the present invention also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0167] like Figure 9 The diagram shows the internal structure of a computer device. This computer device includes a processor, non-volatile storage medium, memory, and a network interface connected via a system bus. The non-volatile storage medium stores the operating system, database, and computer-readable instructions. The database may store control information sequences. When the computer-readable instructions are executed by the processor, they enable the processor to implement a parallel data processing method. The processor provides computational and control capabilities, supporting the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to perform a parallel data processing method. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0168] In this embodiment, the processor is used to execute... Figure 8The specific functions of the recommendation module 110, processing module 120, running module 130, and execution module 140 are described. The memory stores the program code and various types of data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all sub-modules in the parallel data processing device, and the server can call the server's program code and data to execute the functions of all sub-modules.

[0169] In a multi-node computing system, a master node and slave nodes are selected from multiple computing nodes through an election process. The master node receives the target data, performs unified scheduling and task splitting on the target data, and then distributes the split parallel data to the respective slave nodes. The slave nodes perform calculations on the parallel tasks and obtain multiple processing results. The master node merges these multiple processing results to generate the processing result for the target node, and then sends the overall processing result to the corresponding target terminal. This system enables efficient and automated processing of parallel data by allowing the master node to uniformly allocate tasks to the slave nodes.

[0170] The present invention also provides a computer storage medium in which computer-readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the parallel data processing method of any of the above embodiments.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0172] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0173] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A parallel data processing method, characterized in that, include: A master node and at least two slave nodes are elected from a set of predefined computing nodes using a predefined election protocol. The master node parses the target data to be processed, splits the target data into parallel tasks equal to the number of slave nodes, and distributes the parallel tasks to the at least two slave nodes; Each of the at least two slave nodes performs data operations on its respective parallel task, generates a processing result corresponding to the parallel task, and uploads it to the master node. The master node receives the processing results uploaded by each of the at least two slave nodes, merges the processing results, and sends them to a preset target terminal, wherein the target data is uploaded by the target terminal; The step of electing a master node and at least two slave nodes from a preset set of computing nodes through a preset election protocol includes: Collect the second operating parameters of each computing node in the previous data processing cycle; The data of each of the second operating parameters is merged to generate merged data, and the merged data is hashed to generate a hash value; Read the number of nodes in the computing node; Compare the number of nodes with the hash value; When the hash value is less than or equal to the number of nodes, the computing node represented by the node number corresponding to the hash value is determined as the master node, and the other computing nodes are determined as slave nodes. When the hash value is greater than the number of nodes, a remainder value is generated by performing a modulo operation on the hash value based on the number of nodes. The computing node represented by the node number corresponding to the remainder value is determined as the master node, and the other computing nodes are determined as slave nodes.

2. The parallel data processing method according to claim 1, characterized in that, The master node parses the target data to be processed, splits the target data into parallel tasks equal to the number of slave nodes, and distributes the parallel tasks to at least two slave nodes, including: The master node performs data slicing on the target data to generate the parallel tasks, and stores the parallel tasks sequentially in the task queue; The master node generates a first task thread and distributes the parallel tasks in the task queue to the at least two slave nodes through the first task thread.

3. The parallel data processing method according to claim 1, characterized in that, Each of the at least two subordinate nodes performs data computation on its respective parallel task, and each generates a processing result corresponding to the parallel task, including: The main thread of each of the at least two slave nodes starts a task queue and creates a first child thread to continuously receive parallel tasks from the main node. The main thread creates a custom task pool to process the parallel tasks and distributes the parallel tasks to a second child thread that has been initialized and started JTA transactions. The main thread receives the processing result of the parallel task from the second sub-thread and sends the processing result to the main node.

4. A parallel data processing device, characterized in that, include: The election module is used to elect a master node and at least two slave nodes from a set of preset computing nodes through a preset election protocol. The processing module is used by the master node to parse the target data to be processed, split the target data into parallel tasks with the same number of slave nodes, and distribute the parallel tasks to the at least two slave nodes; The computation module is used for each of the at least two slave nodes to perform data computation on their respective parallel tasks, generate processing results corresponding to the parallel tasks, and upload them to the master node. An execution module is used for the master node to receive the processing results uploaded by each of the at least two slave nodes, and to merge the processing results and send them to a preset target terminal, wherein the target data is uploaded by the target terminal; The second acquisition submodule is used to acquire the second operating parameters of each computing node in the previous data processing cycle; The first merging submodule is used to merge the data of each of the second operating parameters to generate merged data, and to perform a hash operation on the merged data to generate a hash value; The second confirmation submodule is used to read the number of computing nodes; compare the number of nodes with the hash value; when the hash value is less than or equal to the number of nodes, determine the computing node represented by the node number corresponding to the hash value as the master node, and determine the other computing nodes as slave nodes; when the hash value is greater than the number of nodes, perform a modulo operation on the hash value according to the number of nodes to generate a remainder value, determine the computing node represented by the node number corresponding to the remainder value as the master node, and determine the other computing nodes as slave nodes.

5. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions that, when executed by the processor, cause the processor to perform the steps of the parallel data processing method as claimed in any one of claims 1 to 3.

6. A computer storage medium, characterized in that, The computer storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the parallel data processing method as described in any one of claims 1 to 3.

Images