Data operation method and device, equipment and storage medium

Abstract

The application discloses a data operation method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method is executed by a first process in multiple processes, and to-be-operated data of the first process comprises multiple first data. Each process in the multiple processes is used for calling an expert network to perform data operation, and the multiple first data can be distributed to processes with a probability greater than a probability threshold for processing. The method comprises the following steps: monitoring the running state of the multiple processes; in response to monitoring a fault process, adjusting the probability of the fault process being distributed to the first data to a value less than the probability threshold, and the first data is used for calling the expert network by the corresponding process to perform data operation. The application can avoid continuing to enable the fault process from the source when the process fails, and does not need to discard the first data previously processed by the fault process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] A Mixture of Experts (MOE) model comprises multiple expert networks, typically invoked in a distributed manner by multiple processes. Each process invokes one expert network. When a process receives input data, it splits the data to be sent to other processes into multiple data blocks. Other processes then invoke their local expert networks to compute the data blocks they are responsible for, and return the results to the original sending process.

[0003] In related technologies, a checkpoint-based restart scheme is provided for the fault tolerance mechanism of multi-process communication in hybrid expert models. When a process fails, the hybrid expert model will be terminated immediately, and the most recent checkpoint data will be loaded to restart all processes.

[0004] However, after restarting all processes, this solution requires reloading tens or even hundreds of gigabytes of model parameters and runtime files, and repeating the running steps before the failure. For large-scale hybrid expert models (such as models containing thousands of expert networks), the recovery time is extremely long, which seriously wastes computing resources. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium, the technical solution of which includes the following contents.

[0006] According to one aspect of this application, a data processing method is provided, which is executed by a first process among multiple processes. The data to be processed by the first process includes multiple first data. Each of the multiple processes is used to call an expert network to perform data processing. The multiple first data can be assigned to processes with a probability greater than a probability threshold for processing. The method includes the following steps.

[0007] Monitor the running status of the multiple processes; In response to the detection of a faulty process, the probability of the faulty process being assigned to the first data is adjusted to a value less than the probability threshold. The first data is used by the corresponding process to call the expert network for data processing.

[0008] According to one aspect of this application, a data processing apparatus is provided, the apparatus running a first process among a plurality of processes, the data to be processed in the first process including a plurality of first data, each of the plurality of processes being used to call an expert network to perform data processing, the plurality of first data being able to be assigned to a process with a probability greater than a probability threshold for processing; the apparatus includes the following modules.

[0009] A monitoring module is used to monitor the running status of the multiple processes; An adjustment module is used to adjust the probability of the faulty process being assigned to the first data to a value less than the probability threshold in response to the detection of a faulty process. The first data is used by the corresponding process to call the expert network for data processing.

[0010] According to one aspect of this application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the data processing method as described above.

[0011] According to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a computer program, the computer program being loaded and executed by a processor to implement the data processing method described above.

[0012] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the data processing method provided in the above aspect.

[0013] The beneficial effects of the technical solutions provided in this application include at least the following:

[0014] The solution proposed in this application reduces the probability of a faulty process being assigned to the first data to below a certain threshold after detecting a faulty process. This prevents the faulty process from being used for data processing from the outset, ensuring that all the first data is distributed among normal processes and processed by the expert network. This solution can ignore faulty processes when they malfunction, guaranteeing uninterrupted processing of all first data and achieving continuous operation of the hybrid expert model. Furthermore, this application eliminates the need to restart processes, discard the first data previously processed by the faulty process, or migrate the first data from the faulty process to the normal process. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the computational architecture of the hybrid expert model provided by related technologies.

[0017] Figure 2 This is a flowchart of a data processing method provided in one embodiment of this application.

[0018] Figure 3 This is a flowchart of a data processing method provided in another embodiment of this application.

[0019] Figure 4 This is a flowchart of a data processing method provided in another embodiment of this application.

[0020] Figure 5 This is a structural block diagram of a data processing device provided in one embodiment of this application.

[0021] Figure 6 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0023] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0024] Hybrid expert models consist of a router and multiple expert networks. The router selects the target expert network from among the multiple expert networks for each input data. For example, the router calculates the probability of routing to each expert network for each input data, thus selecting the expert network with the highest probability as the target expert network and assigning each input data to that network. The expert networks then perform expert computations on the assigned input data. Hybrid expert models decouple the total number of model parameters from the actual computational cost through a sparse activation gating mechanism. Although hybrid expert models have a huge number of parameters, only a small number of expert networks are actually activated during computation, allowing for a very large number of parameters while maintaining controllable computational overhead. For example, hybrid expert models have been successfully applied to ultra-large-scale language models, demonstrating their feasibility at trillions of parameters.

[0025] In a distributed architecture, multiple expert networks are invoked by different processes, and cross-process data distribution and result aggregation need to be completed through all-to-all communication operators. For example, taking the distributed training process of a hybrid expert model as an example, the distributed training process of a hybrid expert model includes the following stages.

[0026] I. Initialization Phase: Based on the preset number of processes for the training task (each process corresponds to one or more expert networks), and by building a fixed all-to-all communication group through the underlying communication library, each process completes registration within the group and clarifies its own rank ID and communication address information.

[0027] 2. Routing Phase: The token sequence generated by the input layer in the hybrid expert model is processed by the router to calculate the probability of each token being assigned to each expert network (corresponding to each process). For a token, the expert network with the highest probability is selected as the target expert, thereby obtaining the routing assignment results for all tokens.

[0028] III. Data Preparation Phase: Each process splits all locally generated tokens into N data blocks (N is the number of processes in the communication group) according to the routing allocation results of the router. At the same time, it initializes the output data buffer to receive tokens sent by other processes.

[0029] IV. All-to-all Communication Phase: Each process sends the split token data block to the corresponding process through the all-to-all communication interface of the communication library, and at the same time receives token data sent by all other processes that need to be processed in the local expert network.

[0030] V. Expert Calculation Phase: Each process transmits the received token to the local expert network to perform calculations and generate calculation results.

[0031] VI. Result Aggregation Stage: The calculation results of each expert are sent back to the original sending process through the reverse all-to-all communication interface to complete the result aggregation and enter the subsequent training process (such as gradient calculation and parameter update).

[0032] In related technologies, the fault tolerance mechanism for all-to-all communication in hybrid expert models still adopts the fault tolerance scheme of general distributed training, mainly including two types: one is a checkpoint-based restart mechanism, which terminates training after a process failure, loads the most recent checkpoint data, and restarts all processes; the other is a process redundancy backup mechanism, which configures backup nodes for critical processes and switches to the backup process when the main process fails. In addition, some improved schemes attempt to trim input and output data or rebuild the communication group after a process failure, but this requires simultaneous adjustment of input and output data sizes and data migration, resulting in high adaptation costs.

[0033] The related technologies have the following drawbacks. First, the fault recovery cost is high and the training efficiency is low. The checkpoint-based restart mechanism requires terminating the training task of the entire hybrid expert model, loading tens or even hundreds of gigabytes of model parameters and training state files, and repeating the training steps before the failure. For large-scale hybrid expert models (such as models containing thousands of experts), the recovery time is extremely long, which seriously wastes computing resources. Second, redundant backup has high resource overhead and poor scalability. The process redundancy backup mechanism requires a backup node for each main process, which continuously occupies resources such as the central processing unit (CPU), graphics processing unit (GPU), and memory. The hybrid expert model itself already requires a lot of resources to deploy the expert network. Redundancy backup will further exacerbate the resource shortage problem. Moreover, as the number of processes increases, the management and synchronization difficulty of backup nodes increases exponentially. Third, the data processing and adaptation are complex and prone to training anomalies. The existing fault tolerance scheme for rebuilding the communication group requires cutting the input and output data corresponding to the faulty process, adjusting the data size, and performing cross-process data migration. This process is prone to causing the token routing mapping relationship to be disordered, and the data migration will generate additional overhead. At the same time, the modification of the input and output data size requires the synchronous adjustment of the calculation parameters of modules such as routers and expert networks. The adaptation logic is complex and is prone to causing a decrease in training accuracy or program crash. Fourth, the router lacks fault adaptation capabilities; the existing router only calculates the allocation probability based on the token characteristics and cannot detect the process fault status. It may still route a large number of tokens to the faulty process, causing these tokens to be unprocessed, which in turn leads to all-to-all communication failure and disrupts the continuity of the training process.

[0034] Figure 1 The diagram illustrates a schematic of the computational architecture of a hybrid expert model provided in an exemplary embodiment of this application. The computational architecture 100 of the hybrid expert model includes multiple processes, each of which is used to invoke an expert network in the hybrid expert model.

[0035] In some embodiments, multiple processes are run by multiple machines, with one process running on each machine. For example, each machine may have one graphics processing unit (GPU) card inserted, and one process will control one GPU card to run an expert network. In other embodiments, at least two processes may be run by the same machine. For example, in a single-machine eight-GPU scenario, one machine may have eight GPU cards inserted, and that machine will run eight processes, with one process controlling one GPU card to run an expert network. In the embodiments of this application, there is no limitation on the correspondence between the number of processes and machines.

[0036] Optionally, the machines include servers, terminal devices, etc. Terminal devices can be at least one of desktop computers, laptops, smartphones, smartwatches, smart TVs, tablets, wearable devices, in-vehicle terminals, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, and MP4 players.

[0037] For example, Figure 1 Taking multiple processes, including process 0, process 1, process 2, and process 3, as an example, a heartbeat mechanism will be deployed between processes 0 and 3 to detect faulty processes. For instance, process 0 will periodically send heartbeat packets to processes 1, 2, and 3. Similarly, process 0 will periodically receive heartbeat packets from processes 1, 2, and 3. Suppose that process 0 does not receive a heartbeat packet from process 1 within a predetermined time, or the information carried in the received heartbeat packet from process 1 is incorrect, process 0 confirms that process 1 is a faulty process. Then, process 0 will synchronize the information that process 1 is a faulty process to processes 2 and 3, so that processes 0, 2, and 3 reach a consensus that process 1 is a faulty process.

[0038] Subsequently, processes 0, 2, and 3 will each reduce the probability of process 1 being assigned the data to be processed during data routing. Taking process 0 as an example, suppose that when process 0 assigns the data to be processed, A, to one of processes 0, 1, 2, or 3, the original probabilities calculated by process 0 for each process are 0.1, 0.4, 0.3, and 0.2 respectively. Then, process 0 will set the original probability corresponding to the faulty process (process 1) as a penalty probability (e.g., -inf), ensuring that the probability score of the faulty process is much lower than that of other normal processes. Then, process 0 will select the process with the highest probability and assign the data to be processed, A, to that process, meaning that the expert network within that process will process the data to be processed. At this point, process 0 will assign the data to be processed to process 2. Through this mechanism, none of the data to be processed by process 0 will be assigned to the faulty process. Similarly, processes 2 and 3 will also not assign their respective data to the faulty process.

[0039] In addition, the temporary master process in processes 0, 2, and 3 (such as the process with the smallest process ID) will re-establish the communication group. For example, the temporary master process will remove the faulty process, collect the communication information of processes 0, 2, and 3, create a communication group based on the communication information of processes 0, 2, and 3 through the communication group creation interface, and assign new temporary process IDs to processes 0, 2, and 3. The temporary master process will then synchronize the communication group information of the re-established communication group to other normal processes.

[0040] Taking process 0 as an example, after distributing all the data to be processed to the various normal processes, process 0 will send the data to be processed allocated to processes 2 and 3 to processes 2 and 3 respectively through the many-to-many communication interface of the communication group. At the same time, process 0 also receives the data to be processed by process 1 sent by processes 2 and 3 through the many-to-many communication interface of the communication group. Process 0 will call the local expert network to calculate all the data to be processed, thereby obtaining the corresponding calculation results, and then send the calculation results back to the original sending process.

[0041] For example, each of the multiple processes will generate two size sets. Taking process 0 as an example, process 0 will generate a first size set based on the amount of data to be processed allocated to processes 2 and 3, and a second size set based on the amount of data to be processed allocated to process 0 by processes 2 and 3. Process 0 will split the data to be processed by processes 2 and 3 into two data blocks based on the first size set, and send the two data blocks to processes 2 and 3 respectively. Also, process 0 will split the calculation results belonging to processes 2 and 3 into two data blocks based on the second size set, and send the two data blocks to processes 2 and 3 respectively.

[0042] In summary, in the embodiments of this application, after detecting a faulty process, each process will reduce the probability of the faulty process being assigned to the data to be processed, thereby avoiding the continued use of the faulty process for data processing from the source, avoiding the interference of the faulty process on cross-process communication, and realizing the continuous operation of the hybrid expert model.

[0043] Figure 2 A flowchart of a data processing method provided by an exemplary embodiment of this application is shown. The method is illustrated by way of example, where the method is executed by a first process. The first process is one of a plurality of processes, each of which is used to call an expert network to perform data processing.

[0044] Step 220: Monitor the running status of multiple processes; For example, the first process monitors the running status of other processes among multiple processes besides itself. Optionally, the running status includes a fault state and a normal state. In this embodiment, a process in a fault state is referred to as a fault process, and a process in a normal state is referred to as a normal process.

[0045] In one embodiment, the first process periodically monitors other processes among multiple processes besides itself. Optionally, the monitoring period is a preset period, such as monitoring other processes once every 10 milliseconds; alternatively, the monitoring period is a dynamically set period based on historical operating conditions. For example, if the historical operating conditions of multiple processes are at a first operating level, the monitoring period value is the first value; or, if the historical operating conditions of multiple processes are at a second operating level, the monitoring period value is the second value. The first value is greater than the second value, and the first operating level is superior to the second operating level. In this case, if the historical operating conditions of multiple processes are good (e.g., no failures for a long time), monitoring other processes with a longer monitoring interval effectively reduces the resources required for monitoring. If the historical operating conditions of multiple processes are poor, monitoring other processes with a shorter monitoring interval ensures that faulty processes can be detected in a timely manner.

[0046] In one embodiment, the first process monitors the other processes among multiple processes sequentially. Optionally, the monitoring order is a preset order, such as monitoring other processes in ascending order of their process identifiers. In this case, the first process monitors the running status of the next process after monitoring the running status of the previous process, ensuring the orderliness of the monitoring and reducing the probability of failure during monitoring.

[0047] In another embodiment, the first process monitors other processes among multiple processes in parallel. The first process will monitor the other processes among multiple processes simultaneously through distributed parallel monitoring. In this case, the distributed parallel monitoring method can improve monitoring efficiency.

[0048] In one embodiment, the first process monitors other processes by reading the running status identifiers of other processes among multiple processes. For example, each process maintains a running status identifier. When a process fails, the status monitoring subprocess of that process updates the running status identifier of that process to the fault identifier. When the first process reads the fault identifier of a process, it determines that the process is a faulty process.

[0049] At this point, by indirectly monitoring processes by reading the running status identifier, the first process can quickly learn about the running status of other processes. Furthermore, since the reading resources required by the first process to read the running status identifier and the maintenance resources required by each process to maintain the running status identifier are relatively low, the monitoring cost of the first process monitoring other processes is effectively controlled.

[0050] Step 240: In response to the detection of a faulty process, the probability of the faulty process being assigned to the first data is adjusted to a value less than the probability threshold. The first data is used by the corresponding process to call the expert network for data processing.

[0051] The data to be processed in the first process includes multiple first data items. These multiple first data items can be assigned to processes with a probability greater than a probability threshold for processing.

[0052] Optionally, the probability threshold is a pre-set minimum value, such as a probability threshold of 0.01; or the probability threshold is a dynamically set threshold, such as the probability threshold being the minimum probability that all normal processes are assigned to the first data.

[0053] For example, in response to detecting a faulty process, the probability of the faulty process being assigned to the first data is reduced to a value less than the probability threshold, so that multiple first data are assigned to normal processes other than the faulty process. Each normal process will call its own expert network to perform data computation based on the assigned first data.

[0054] For example, the first process will use a router to reduce the probability of a faulty process being assigned to the first data, so that each of the multiple first data is assigned to each of the normal processes other than the faulty process, and any two normal processes are assigned different first data. Optionally, the router is a gated network in a hybrid expert model.

[0055] For example, multiple first data points are multiple first tokens. These first tokens are obtained by the first process calling the input layer of the hybrid expert model to segment the original input data. The original input data is the input data of the hybrid expert model. The original input data is the data received by the first process.

[0056] For example, the multiple first data points are data generated during the training process of the hybrid expert model. In this case, the data processing method provided in this application can be applied to the distributed training of the hybrid expert model. Optionally, the multiple first data points are data generated during the inference process of the hybrid expert model. In this case, the data processing method provided in this application can be used in the distributed inference of the hybrid expert model. The data processing method provided in this application can be applied to the training or inference process of the hybrid expert model, and this application does not limit it.

[0057] For example, the first process calls the router to calculate the probability of each piece of first data being assigned to each of the multiple processes. Then, it reduces the probability of a faulty process being assigned to the first data, so that when assigning the process corresponding to the first data based on probability, each piece of first data will not be assigned to a faulty process.

[0058] It is understandable that allocating the first data process based on probability is an existing step in the multi-process distribution scheme of the hybrid expert model. This application will reduce the probability of the faulty process being allocated the first data from the source when a faulty process is detected, thereby organically combining the existing probability allocation steps to avoid allocating the first data to the faulty process, so that the faulty process does not have any data to transmit, and thus the faulty process will naturally not participate in the subsequent inter-process communication process.

[0059] With the solution proposed in this application, the faulty process will not be allocated the first data from the source. All the first data will be divided among other normal processes, which can continue to execute the communication process, thus successfully allocating all the first data to the expert network for computation. The solution provided in this application can ignore the faulty process when it malfunctions, ensuring that all the first data can continue to be processed without interruption.

[0060] In addition, the solution provided in this application has strong scalability and adaptability. The solution can handle scenarios of single or multiple process failures and is applicable to hybrid expert models of different scales (from hundreds of experts to thousands of experts). As the number of processes increases, the fault tolerance overhead grows slowly, and the scalability is excellent.

[0061] based on Figure 2 In the optional embodiment shown, step 240, "in response to detecting a faulty process, adjusting the probability of the faulty process being assigned to the first data to a value less than a probability threshold," includes... Figure 3 Steps 241 to 242 are shown.

[0062] Step 241: In response to the detection of a faulty process, calculate the original probability set, which includes the original probability of each of the multiple first data points being assigned to each of the multiple processes. For example, the raw probability is the probability generated during the process of the first process calling the router to distribute the first data to various processes among multiple processes. Optionally, the raw probability is the probability before the normalization operation.

[0063] For example, the original probability set includes multiple probability subsets, each probability subset corresponding to a first data point, and each probability subset includes the original probability of assigning the first data point to each of the multiple processes; for example, one probability subset in the original probability set is: first data A [-0.8, 0.4, 1.2, 0.6]; wherein, the probability subset indicates that the original probability of first data A being assigned to the first process is -0.8, the probability of first data A being assigned to the second process is 0.4, the probability of first data A being assigned to the third process is 1.2, and the probability of first data A being assigned to the fourth process is 0.6.

[0064] Step 242: In the original probability set, the original probability corresponding to the faulty process is reduced to the penalty probability to obtain the modified probability set, wherein the penalty probability corresponding to the faulty process in each probability subset of the modified probability set is less than the probability threshold. For example, the corrected probability set refers to the probability set obtained by lowering the original probabilities of all faulty processes in the original probability set to their penalty probabilities. In each subset of the corrected probability set, the penalty probability of a faulty process is less than the original probability of any normal process.

[0065] For example, continuing with the probability subset example above, assuming the third process is a faulty process, the first process will set the original probability corresponding to the third process as a penalty probability, resulting in the corrected probability subset: the first data A [-0.8, 0.4, penalty probability, 0.6]; similarly, for each probability subset, the original probability corresponding to the faulty process is set as a penalty probability, such that the penalty probability corresponding to the faulty process in each probability subset is less than the original probability corresponding to any normal process.

[0066] Optionally, in the original probability set, the original probability corresponding to the faulty process is set to negative infinity (-inf) to obtain the corrected probability set.

[0067] Optionally, a modified probability set can be obtained by multiplying the original probability corresponding to the faulty process by a penalty factor in the original probability set. Optionally, if the original probability corresponding to the faulty process is negative, the original probability is multiplied by a penalty factor, where the first penalty factor is greater than 1 (e.g., inf). Optionally, if the original probability corresponding to the faulty process is positive, the original probability is multiplied by a second penalty factor, where the second penalty factor is less than 1 (e.g., -inf). Optionally, the penalty factor is a preset value. Optionally, the penalty factor can be dynamically adjusted according to computational needs. For example, if the data precision of the hybrid expert model is high, a higher precision penalty factor (e.g., 0.0001) can be set; conversely, if the data precision of the hybrid expert model is low, a lower precision penalty factor (e.g., 0.01) can be set.

[0068] After obtaining the corrected probability set, the process corresponding to the highest probability in each probability subset will be used as the target routing process, and the first data corresponding to each probability subset will be allocated to the target routing process.

[0069] It should be noted that the step of selecting the process with the highest probability as the target routing process is a step that is already present in the original router's allocation logic. This embodiment only changes the original probability corresponding to the faulty process, and then reuses the router's existing step of "selecting the process with the highest probability as the target routing process", thus achieving an organic combination of existing steps and reducing the cost required to achieve the desired effect.

[0070] Continuing with the explanation of the probability subset above, the first process will select the fourth process with the highest probability from the probability subset: first data A [-0.8, 0.4, penalty probability, 0.6] as the target routing process for first data A.

[0071] Optionally, the first process performs a normalization operation, such as a softmax operation, on the modified probability set to obtain a normalized probability set. For the normalized probability set, the process corresponding to the highest probability in each probability subset is taken as the target routing process, and the first data corresponding to each probability subset is allocated to the target routing process.

[0072] Using the probability subset mentioned above, where the penalty probability is -inf, the probability subset: First data A [-0.8, 0.4, -inf, 0.6] will be normalized to obtain the probability subset: First data A [0.1194, 0.3964, 0.0000, 0.4842]. Then, the first process will select the fourth process with the highest probability as the target routing process for the first data A.

[0073] In related technologies, the router calculates the original probability of each first data point being assigned to each of the multiple processes, and then performs a normalization operation on all the original probabilities to obtain all the normalized probabilities. For the first data A, the related technologies select the process with the highest probability from all the normalized probabilities corresponding to the first data A, take the process with the highest probability as the target routing process, and assign the first data A to the target routing process.

[0074] In this embodiment, the router calculates the original probability of each first data point being assigned to each of the multiple processes. Then, it modifies the original probability corresponding to the faulty process to a penalty probability (e.g., -inf) to obtain a corrected probability set. Next, this embodiment performs a normalization operation on the corrected probability set to obtain all normalized probabilities. For the first data A, this embodiment selects the process with the highest probability from all the normalized probabilities corresponding to the first data A, uses the process with the highest probability as the target routing process, and assigns the first data A to the target routing process.

[0075] In other words, in this embodiment, after generating the original probability and before normalizing the probability, the original probability corresponding to the faulty process is modified into a penalty probability. Other steps reuse related technologies. This embodiment only requires a small degree of code intrusion to achieve the continuous operation of the hybrid expert model.

[0076] In this embodiment, by setting a penalty probability for the faulty process, the probability of the faulty process being assigned the first data is extremely low, thereby preventing the first data from being routed to the faulty process from the source and avoiding invalid communication. In addition, the above embodiment also provides two schemes for setting the penalty probability for the faulty process: setting the original probability corresponding to the faulty process to negative infinity and multiplying it by a penalty factor. The penalty probability can be set for the faulty process by setting the value or multiplying by the penalty factor, which simplifies the setting operation of the penalty probability.

[0077] Furthermore, it is understood that the above embodiments only suppress the probability of faulty processes, so that the routing logic and probability distribution of normal processes are not affected, thereby maintaining the original routing characteristics; in addition, the above embodiments also ensure that no invalid data to be calculated participates in the calculation, ensuring the validity of the expert calculation results.

[0078] based on Figure 2 In the optional embodiment shown, step 220 includes: monitoring the running status of multiple processes through a heartbeat interaction mechanism.

[0079] A heartbeat mechanism refers to a mechanism in which multiple processes periodically send heartbeat packets to each other. A heartbeat packet is a data packet used to confirm whether a process has failed. Optionally, the period for sending heartbeat packets between multiple processes is 10 milliseconds, with each process sending a heartbeat packet to other processes every 10 milliseconds. For example, a first process uses the heartbeat mechanism to monitor whether other processes besides itself have failed.

[0080] In one embodiment, for a target process, in response to not receiving a heartbeat packet from the target process within a preset time or receiving a heartbeat packet from the target process containing incorrect information, the first process determines the target process as a faulty process; wherein, the target process is any process other than the first process among a plurality of processes.

[0081] When no other processes besides the first process have failed, the first process will receive heartbeat packets from each of the other processes within a preset time. Each heartbeat packet contains correct information. Therefore, if the first process does not receive a heartbeat packet from a certain process within the preset time, or if the heartbeat packet from a certain process contains incorrect information, the first process can determine that the process has failed.

[0082] For example, the first process maintains a timeout counter. The first process uses the timeout counter to keep track of time. If the first process does not receive a heartbeat packet from the target process within a preset time, the target process is determined to be a faulty process.

[0083] Optionally, the preset time is the time used to determine process failure, such as 50 milliseconds.

[0084] Optionally, the heartbeat packet includes at least one of the following: process identifier, current stage information, and resource usage status information. Optionally, the current stage information includes at least one of the following: routing calculation stage, inter-process communication stage, and expert calculation stage. Optionally, the resource usage status information includes at least one of the following: CPU utilization, GPU utilization, and memory usage.

[0085] For example, if the heartbeat packet received from the target process contains at least one of resource overflow information or program exception information, the first process determines that the target process is a faulty process.

[0086] Optionally, the first process also marks the target process as an unavailable process in the fault process information so that the target process can be marked as a fault process in subsequent fault process information.

[0087] Optionally, in response to detecting a faulty process, the first process synchronizes the faulty process information to the normal process via broadcast; wherein the faulty process information includes at least one of the following: the process identifier of the faulty process, the fault type, and the time of the fault occurrence.

[0088] Optionally, the faulty process information includes process identifiers of multiple processes, as well as unavailable identifiers used to mark one or more processes as faulty processes, so as to achieve the above-mentioned faulty process information including the process identifiers of faulty processes.

[0089] For example, the first process will synchronize the fault information of the detected faulty process to other normal processes to ensure that all normal processes have a consistent understanding of the faulty process. This will enable the first process and other normal processes to reduce the probability of the faulty process when the router allocates data to be processed, thereby avoiding the allocation of data to the faulty process.

[0090] In this embodiment, the first process will monitor other processes through a heartbeat interaction mechanism. The heartbeat interaction mechanism can quickly identify faulty processes, laying the foundation for subsequent operations to reduce the probability of faulty processes.

[0091] based on Figure 2 In the optional embodiment shown, the first process is a temporary main process, and the number of normal processes among the multiple processes is M, where M is a positive integer greater than one. The method further includes... Figure 4 Steps 420 and 440 are shown. Optionally, since the first process is the process with the smallest process ID among the M normal processes, the first process will establish the communication group as a temporary master process.

[0092] Step 420: Establish a communication group based on M normal processes; Before the M normal processes can communicate with each other, a communication group needs to be built. The communication group contains the M normal processes. Processes in the communication group can communicate with each other. Building a communication group can isolate faulty processes and speed up the communication efficiency between the M normal processes.

[0093] In one embodiment, a first process collects communication information from M normal processes; based on the communication information, an initial communication group is created through a communication group creation interface; and a process identifier is reassigned to each normal process in the initial communication group to obtain the communication group.

[0094] For example, the communication information of a normal process includes at least one of the normal process's communication address (such as IP address, port number) and process identifier.

[0095] For example, the communication group creation interface includes at least one of the MPI_Comm_create and NCCLCommInitRank function interfaces. The MPI_Comm_create function interface is a function interface in the MPI (Message Passing Interface) library used to create communication groups. It receives a specific group of processes and returns a new communication group; only processes belonging to that group can call the MPI_Comm_create function interface to create the communication group. The NCCLCommInitRank function interface is used to create an internal connection for a specified graphics processor to obtain a communication group.

[0096] For example, the communication information of all normal processes is input into the communication group creation interface so that the communication group creation interface can create a communication group containing M normal processes.

[0097] For example, the first process reassigns process identifiers to each normal process in the initial communication group in increments from zero to M-1. Optionally, the first process also maintains a mapping table between the original process identifiers and the reassigned process identifiers of each process in the communication group.

[0098] In this embodiment, a method for rebuilding the communication group is provided. Since the faulty process has been prevented from being assigned to the first data from the source, the communication group can be rebuilt using all the normal processes. The faulty process can be ignored when rebuilding the communication group.

[0099] In one embodiment, the first process also synchronizes the communication group information of the communication group to other normal processes, so that the other normal processes update their local communication group configuration based on the received communication group information, thereby ensuring that the first process and other normal processes have a consistent understanding of the new communication group.

[0100] Optionally, the communication group information includes at least one of the following: the original process identifier of each normal process in the communication group, the reassigned process identifier, and the size information of the communication group. Optionally, the size information of the communication group refers to the number of processes contained in the communication group.

[0101] In one embodiment, the first process also needs to configure communication parameters for the communication group. Optionally, the first process configures the communication parameters for the communication group based on the size of the communication group and the target size set; The communication parameters include the address of the first process's send buffer, the address of the receive buffer, the target size set, and the communication timeout.

[0102] For example, after the first process builds the communication group, it will know the size of the communication group and configure the address of the send buffer, the address of the receive buffer, and the communication timeout for the communication group according to the size of the communication group.

[0103] The send buffer is used to cache the data to be processed by the first process. The receive buffer is used to cache the data to be processed received by the first process.

[0104] For example, if the communication group is large, that is, if there are many processes in the communication group, the first process is configured with a wider range of send buffer addresses, a wider range of receive buffer addresses, and a longer communication timeout period to match the size of the communication group; wherein, the communication timeout period is used to indicate the time for determining timeout, and when the communication between two processes exceeds the communication timeout period, it indicates that the communication is abnormal.

[0105] For example, the first process generates a target size set; the target size set includes a first size set and a second size set, the first size set including the amount of first data allocated to each of the M-1 normal processes, and the second size set including the amount of data to be processed allocated to the first process by each of the M-1 normal processes. The generation process of the first size set and the second size set will be described below.

[0106] Step 440: Execute the communication process with M-1 normal processes through the many-to-many communication interface of the communication group.

[0107] M-1 normal processes include all processes other than the first process among the M normal processes.

[0108] For example, the many-to-many communication interface includes at least one of the MPI_alltoall interface and the NcclAllToAll interface. The MPI_alltoall interface belongs to the MPI communication library and is used to implement fully switched communication, meaning each process can send different data to other processes while simultaneously receiving data from all other processes. The NcclAllToAll interface is used to implement fully switched communication between graphics processors for efficient transfer of tensor data in the graphics processor's video memory. It is understood that in this embodiment, without relying on a specific communication library, any type of communication library can be used with the fault-tolerant scheme provided in this application.

[0109] In one embodiment, the first process obtains the target size set; based on the target size set, it performs a communication process with M-1 normal processes through the many-to-many communication interface of the communication group.

[0110] The target size set includes at least one of a first size set and a second size set; the first size set includes M-1 first sizes, each first size indicating the amount of first data allocated to a normal process. Optionally, the first process generates the first size set based on the amount of first data allocated to each of the M-1 normal processes; for example, if the router's allocation result is: the first process is allocated 100 pieces of first data, the second process is allocated 300 pieces of first data, the third process is allocated 400 pieces of first data, and the fourth process is allocated 200 pieces of first data, then the first size set generated by the first process (without recording the amount of data allocated to the first process) is [300, 400, 100].

[0111] The second size set includes M-1 second sizes, each indicating the amount of data to be processed allocated to the first process by a normal process. Optionally, the first process generates the second size set based on the amount of data to be processed allocated to it by each of the M-1 normal processes; for example, if the second process allocates 100 data to be processed to the first process, the third process allocates 200 data to be processed to the first process, and the fourth process allocates 400 data to be processed to the first process, then the second size set is [100, 200, 400]. Optionally, each normal process other than the first process will synchronize the amount of data allocated to the first process to the first process, and the first process will summarize the received data to obtain the second size set.

[0112] In this embodiment, after rebuilding the communication group, there is no need to change the size of the data to be processed output and input by each process, nor is there a need to relocate the data of the faulty process (because the faulty process will not be allocated data). Instead, the data block size can be dynamically adjusted according to the router allocation result to adapt to the new communication group size, which reduces the program development and maintenance costs and reduces the possibility of training anomalies.

[0113] In one embodiment, the first process, based on a target size set, performs a communication process with M-1 normal processes, including at least one of the following processes: I. The target size set includes the first size set; The first process divides the remaining first data into M-1 first data blocks according to the first size set; the remaining first data is the first data other than the first data allocated by the first process. Through the many-to-many communication interface of the communication group, M-1 first data blocks are sent to M-1 normal processes; Each of the M-1 first data blocks contains several first data items and corresponds to a normal process. The first size set contains the sizes of the M-1 first data blocks.

[0114] For example, the first process generates M-1 first data blocks according to the first size set; then, the other first data, except for the first data allocated to itself, is divided into the M-1 first data blocks according to the router allocation result.

[0115] In related technologies, the step of dividing data blocks according to a set of data block sizes is also performed. In this embodiment, only the first size set needs to be dynamically generated to adapt to the newly established communication group, so that the step of "dividing data blocks according to a set of data block sizes" in related technologies can be reused. This embodiment has less intrusiveness to the code, and only minor modifications are needed to ensure that the hybrid expert model does not stop running when there is a process failure.

[0116] Optionally, the first process also receives M-1 second data blocks sent by M-1 normal processes through the many-to-many communication interface of the communication group; Integrity checks are performed on M-1 second data blocks using the first size set; Among them, each of the M-1 second data blocks contains several first calculation results. These several first calculation results are obtained by a normal process calling the expert network to perform calculations on the received several first data.

[0117] For example, the M-1 second data blocks are obtained by each of the M-1 normal processes by calling the expert network to perform calculations based on the received first data blocks; each of the M-1 normal processes will call the local expert network to perform calculations on each first data in the received first data block to obtain the second calculation result.

[0118] For example, the first process will perform integrity checks on M-1 second data blocks according to the first size set to ensure that the size of each second data block is consistent with the first size recorded in the first size set, thereby ensuring that each second data block has no missing or redundant data.

[0119] II. The target size set includes the second size set; According to the second size set, the first process divides the calculation results of the local expert network of the first process, which belong to M-1 normal processes, into M-1 third data blocks; M-1 third data blocks are sent to M-1 normal processes through the many-to-many communication interface of the communication group; Among them, each of the M-1 third data blocks contains several second calculation results. The several second calculation results are obtained by the first process calling the local expert network to perform calculations on several data to be calculated in a fourth data block. A fourth data block contains several data to be calculated that are allocated to the first process by a normal process.

[0120] For example, after the first process receives M-1 fourth data blocks, it will call the local expert network to perform calculations for each piece of data to be calculated in each fourth data block, and obtain the calculation result corresponding to each piece of data to be calculated in each fourth data block.

[0121] For example, the first process divides the calculation results of the local expert network, which belong to M-1 normal processes, into M-1 third data blocks. Each third data block contains several first calculation results. Furthermore, the receiving process of the target third data block is consistent with the sending process of the target second data block. The target third data block is obtained by performing calculations on several data to be calculated in the target fourth data block. The target third data block is any one of the M-1 third data blocks.

[0122] For example, the first process generates M-1 third data blocks according to the second size set; the second size set records the sizes of the M-1 fourth data blocks allocated to the first process by the M-1 normal processes, so the first process can use the second size set again to generate M-1 third data blocks; Then, the first process divides the calculation results of the local expert network, which belong to the M-1 normal processes, into the M-1 third data blocks according to the source paths of the M-1 fourth data blocks.

[0123] In related technologies, the step of dividing data blocks according to a set of data block sizes is also performed. In this embodiment, only the second size set needs to be dynamically generated to adapt to the newly established communication group, so that the step of "dividing data blocks according to a set of data block sizes" in related technologies can be reused. This embodiment has less intrusiveness to the code, and only minor modifications are needed to ensure that the hybrid expert model does not stop running when there is a process failure.

[0124] Optionally, the system can receive M-1 fourth data blocks sent by M-1 normal processes through the many-to-many communication interface of the communication group. Integrity checks are performed on M-1 fourth data blocks using the second size set; Among them, each of the M-1 fourth data blocks contains several data to be processed that were allocated to the first process by a normal process.

[0125] For example, the first process needs to receive data to be processed sent to it by other normal processes. The first process will receive a fourth data block sent by each of the M-1 normal processes, and each fourth data block contains several pieces of data to be processed allocated to the first process by a normal process.

[0126] For example, the first process will perform integrity checks on M-1 fourth data blocks according to the second size set to ensure that the size of each fourth data block is consistent with the second size recorded in the second size set, thereby ensuring that each fourth data block has no missing or redundant data.

[0127] In this embodiment, data blocks or verification data blocks are split out using dynamically adjusted first size sets and / or second size sets, thereby enabling smooth cross-process communication within the communication group after ignoring faulty processes. This embodiment does not require changing the scale of the data to be processed output and input by each process, nor does it require relocating the data of faulty processes. Only by dynamically adjusting the size of the data blocks, the original data block partitioning steps and data block sending steps can be reused to complete cross-process communication within the communication group.

[0128] Optionally, the first process will also aggregate the local computation result block with M-1 second data blocks to obtain the data aggregation result; The local computation result block contains the computation results of the first process calling the local expert network and performing calculations based on several allocated first data; each of the M-1 second data blocks is obtained by a normal process calling the expert network and performing calculations on several first data.

[0129] Optionally, the first process uses methods such as token-dimensional concatenation and gradient-weighted summation to aggregate the local computation result block with M-1 second data blocks to obtain the data aggregation result. Optionally, the first process performs subsequent operations based on the data aggregation result, such as loss calculation and parameter updates. If a new faulty process is subsequently detected, steps 220 and 240 are repeated to repeat the fault-tolerant process and ensure that the computation process is not interrupted.

[0130] In some embodiments, a data processing method is provided, illustrated by the example of the method being executed by multiple processes. The method includes the following steps S1 to S6.

[0131] Step S1: Each of the multiple processes sends a heartbeat packet to other processes at a preset period. If a process does not receive a heartbeat packet from the target process within a preset timeout period, or if the received heartbeat packet contains error information, the target process is determined to be a faulty process.

[0132] Optional, the preset period is 10 milliseconds; optional, the preset timeout is 50 milliseconds.

[0133] Optionally, the heartbeat packet may include information such as process rank ID, current stage information (e.g., routing calculation stage, all-to-all communication stage, expert calculation stage), and resource usage status information (e.g., CPU utilization, GPU utilization, memory usage).

[0134] Optionally, each process maintains a heartbeat reception timeout counter to determine whether a heartbeat packet has timed out.

[0135] Optionally, when a faulty process is detected by a discovery process among multiple processes, the discovery process will broadcast the faulty process information (including the faulty process's rank ID, fault type, and fault occurrence time) to all normal processes to ensure that all normal processes have a consistent understanding of the faulty process.

[0136] In step S2, each normal process dynamically modifies the router's probability calculation logic based on the synchronized faulty process information to prevent token routing to the faulty process.

[0137] Optionally, each normal process can preset a probability penalty factor (e.g., directly set -inf, which can be dynamically adjusted according to training needs). Upon receiving information about a faulty process, a penalty is applied to the probability calculation branch corresponding to each faulty process. For example, in the original probability of each token being allocated to each process, each normal process sets the original probability corresponding to the faulty process to -inf to obtain the corrected probability, ensuring that the score corresponding to the faulty process is much lower than that of the normal process; optionally, the original probability is the probability before normalization.

[0138] For example, each normal process first performs a softmax normalization operation based on the corrected probability, and then performs token routing allocation. At this time, processes with a probability below the threshold (i.e., faulty processes) will be automatically removed, and the process with the highest probability will be selected from the remaining normal processes to ensure that no tokens are routed to the faulty process.

[0139] Step S3: The temporary master process rebuilds the communication group based on the remaining normal processes by removing the faulty processes.

[0140] Optionally, the temporary master process filters out the normal processes that are in an available state based on the faulty process information, forming a set of normal processes S, the size of the set of normal processes S is M (M = the number of original processes N - the number of faulty processes K); optionally, the temporary master process is the process with the smallest rank ID.

[0141] Optionally, the temporary main process performs communication group initialization operations. For example, the temporary main process collects information such as the communication addresses (IP addresses, port numbers) and process identifiers of all normal processes, creates a new communication group through the communication library creation interface, and then assigns a new temporary rank ID to each normal process (the new rank ID increments sequentially from 0 to M-1, forming a mapping table with the original rank ID).

[0142] Optionally, the temporary master process synchronizes the member information of the new communication group (including the original rank ID, temporary rank ID, and communication group size M of the normal processes) to all normal processes, and each normal process updates its local communication group configuration to complete the establishment of the new communication group.

[0143] Step S4: Each normal process generates an input size set and an output size set.

[0144] Taking the first process as an example, other normal processes are implemented similarly. The first process will generate its own set of input dimensions and set of output dimensions; for example, the set of input dimensions generated by the first process includes the data block size received by each normal process when the first process distributes tokens to each normal process; the set of output dimensions generated by the first process includes the data block size distributed by each normal process to the first process.

[0145] For example, since the aforementioned steps have ensured that no token is routed to the faulty process, the original data block corresponding to the faulty process is empty. Therefore, the overall size of the input / output data of each normal process does not need to be modified, and each normal process only needs to generate the data block size according to the routing allocation result.

[0146] Step S5: Each normal process communicates based on the communication group.

[0147] For example, each normal process communicates on the newly created communication group based on the input size set and the output size set. The specific process is as follows: 1. Communication parameter configuration: Each normal process configures communication parameters based on the size M of the new communication group, the set of input sizes, and the set of output sizes. Communication parameters include send / receive buffer addresses, data block sizes, communication timeouts, etc. 2. Data transmission and reception: Each normal process adapts to the many-to-many communication interface of the new communication group through the communication library. According to the input size set, it sends the M data blocks after splitting the input data to all normal processes in the new communication group, and at the same time receives the data blocks sent by all normal processes (the faulty process has no data to send, and the corresponding receive block is empty). 3. Communication result verification: Each normal process performs integrity verification on the received data blocks (such as verifying data length and data checksum) to ensure that the received data blocks match the output size set and that there is no missing or redundant data.

[0148] In step S6, each normal process calls the local expert network to perform calculations.

[0149] After communication is complete, each normal process performs expert calculations and aggregates the results, as detailed below: 1. Expert computation execution: Each normal process transmits the received token data (from the normal process) to the local expert network. Since the router has ensured that the data are all valid tokens adapted to the local expert network, no additional data filtering is required. It directly performs tasks such as feature transformation and gradient calculation to generate local computation results. 2. Data aggregation process: Based on the communication group and the output size set, reverse all-to-all communication is initiated—each normal process splits the local expert calculation results according to the output size set and sends the split results to the original sending process; at the same time, it receives the calculation results sent by other normal processes, combines and processes them according to preset aggregation rules (such as token dimension concatenation, gradient weighted summation) to generate a globally unified aggregation result; 3. Training process continuation: The aggregated results will be used to perform subsequent training steps (such as loss calculation and parameter update); if a new process failure is detected later, the above fault-tolerant process will be repeated to ensure that training is not interrupted.

[0150] The following is a specific implementation method to more clearly illustrate the solution proposed in this application.

[0151] In one possible implementation, taking the distributed training scenario of the Transformer-MoE model based on the NCCL communication library as an example, it is assumed that the original training task uses four processes (rank 0~rank 3, each process deploying one expert network) to build a communication group to achieve cross-process distribution of tokens and expert computation. The specific implementation steps are as follows: 1. Initial state settings Each process initializes the original communication group based on the NCCL communication library. Four processes (rank 0 to rank 3) complete the registration within the group. It is assumed that the length of the token data processed by each process is 1000. Each process sends heartbeat packets to other processes at a period of 10 milliseconds, and the timeout is set to 50 milliseconds.

[0152] 2. Fault detection and status synchronization Suppose that during the process of the router performing token probability calculation (i.e. token allocation), the rank 2 process crashes due to a memory access anomaly and stops sending heartbeat packets; The other three processes (rank 0, rank 1, and rank 3) did not receive a heartbeat packet from the rank 2 process after a 50-millisecond timeout, so the rank 2 process was determined to be a faulty process and marked as "unavailable".

[0153] 3. Reset probability distribution The probability tensor for 1000 tokens has a shape of [1000, 4] (assuming there are 4 expert networks, one running on each graphics processing unit card). Taking one token as an example: The token is initially assigned to one of the four expert networks with the following possible values: [-0.8, 0.4, 1.2, 0.6]. After softmax normalization, the value is [0.0634, 0.2106, 0.4687, 0.2572]. At this point, the token will be routed to the rank2 process.

[0154] When the rank2 process fails and exits training, the possible values ​​of the token assigned to the four expert networks will be updated to [-0.8, 0.4, -inf, 0.6]. After softmax normalization, the values ​​will be [0.1194, 0.3964, 0.0000, 0.4842]. At this point, the token will be routed to the rank3 process instead of the rank2 process.

[0155] 4. Dynamically adjust and change the split size Assuming the routing mechanism is calculated using the original probability, the number of tokens allocated to each process out of the 1000 tokens is as follows: Rank0: [300, 100, 400, 200]; Rank1: [400, 200, 300, 100]; Rank2: [200, 400, 200, 200]; Rank3: [100, 200, 300, 400]; In this context, Rank 0 means that of the 1000 tokens allocated to process 0, 300 are allocated to process 0, 100 to process 1, 400 to process 2, and 200 to process 3. The remaining Ranks 1 to 3 can be referred to in the description of Rank 0, and will not be repeated here.

[0156] The number of these tokens constitutes the input split size, where the number of tokens recorded in Rank0 is the first size set; Correspondingly, the number of tokens each process receives from all other processes is: Rank0: [300, 400, 200, 100]; Rank1: [100, 200, 400, 200]; Rank2: [400, 300, 200, 300]; Rank3: [200, 100, 200, 400]; In this context, Rank 0 represents that process 0 received 300 tokens from process 0, 400 tokens from process 1, 200 tokens from process 2, and 100 tokens from process 3. The remaining Ranks 1 to 3 can be referred to in the same way as Rank 0, and will not be repeated here.

[0157] The number of these tokens constitutes the output split size, where the number of tokens recorded in Rank0 is the second size set.

[0158] When the rank2 process encounters an error, its probability is reset to 0, and no tokens will be distributed to it. The tokens previously allocated to the rank2 process will then be unevenly distributed to other processes. The inputs split size might then be: Rank0: [500, 200, 0, 300]; Rank1: [500, 300, 0, 200]; Rank3: [150, 300, 0, 550]; The output split size now becomes: Rank0: [500, 500, 0, 150]; Rank1: [200, 300, 0, 300]; Rank3: [300, 200, 0, 550]; At this point, the number of tokens corresponding to the rank2 process needs to be removed to form new_input_split_size: Rank0: [500, 200, 300]; Rank1: [500, 300, 200]; Rank3: [150, 300, 550]; At this point, the output split size becomes new_output_split_size: Rank0: [500, 500, 150]; Rank1: [200, 300, 300]; Rank3: [300, 200, 550]; 5. Communication group reconstruction Communication group reconstruction: Normal processes (rank 0, 1, 3) form a set S (size M=3). The rank 0 process acts as a temporary master process, calls the ncclCommInitRank interface to create a new communication group, and assigns a new temporary rank ID. For example, the original rank 0 process is changed to the new rank 0 process, the original rank 1 process is changed to the new rank 1 process, and the original rank 3 process is changed to the new rank 2 process. The communication group information is synchronized to all normal processes.

[0159] 6. Conduct communication under the newly established communication group. In the newly created communication group, the input and output data sizes of each process remain the same (1000 tokens for input and 1000 tokens for output). Only input_split_size is replaced with new_input_split_size and output_split_size is replaced with new_output_split_size. Then, normal all-to-all operator communication can be called.

[0160] Figure 5 This application illustrates a data processing apparatus 500 provided in an exemplary embodiment. The apparatus operates a first process among multiple processes. The data to be processed in the first process includes multiple first data sets. Each process in the multiple processes is used to invoke an expert network to perform data processing. The multiple first data sets can be assigned to processes with probabilities greater than a probability threshold for processing. The apparatus 500 includes: Monitoring module 501 is used to monitor the running status of multiple processes; The adjustment module 502 is used to adjust the probability of the faulty process being assigned to the first data to a value less than the probability threshold in response to the detection of a faulty process. The first data is used by the corresponding process to call the expert network for data processing.

[0161] In an optional embodiment, the adjustment module 502 is configured to calculate an original probability set in response to detecting a faulty process. The original probability set includes the original probability of each of the multiple first data points being assigned to each of the multiple processes. In the original probability set, the original probability corresponding to the faulty process is reduced to a penalty probability to obtain a modified probability set. In each probability subset of the modified probability set, the penalty probability corresponding to the faulty process is less than the probability threshold. Each probability subset contains the probability of allocating a first data point to each of the multiple processes.

[0162] In an optional embodiment, the adjustment module 502 is used to set the original probability corresponding to the faulty process to negative infinity in the original probability set to obtain a modified probability set. or, In the original probability set, the original probability corresponding to the faulty process is multiplied by the penalty factor to obtain the corrected probability set. The penalty factor is greater than 1 when the original probability corresponding to the faulty process is negative, and less than 1 when the original probability corresponding to the faulty process is positive.

[0163] In an optional embodiment, the monitoring module 501 is used to monitor the running status of multiple processes through a heartbeat interaction mechanism.

[0164] In an optional embodiment, the monitoring module 501 is configured to determine that the target process is a faulty process in response to the fact that no heartbeat packet of the target process is received within a preset time or the information contained in the heartbeat packet of the target process is incorrect. The target process is any process other than the first process among multiple processes.

[0165] In an optional embodiment, the device further includes a synchronization module 503. The synchronization module 503 is configured to, in response to detecting a faulty process, synchronize the faulty process information of the faulty process to the normal process via broadcast. The fault process information includes at least one of the following: the process identifier of the fault process, the fault type, and the time of fault occurrence.

[0166] In an optional embodiment, the first process is a temporary main process, and the number of normal processes among the multiple processes is M. The device also includes a communication group establishment module 504 and a communication module 505.

[0167] Communication group establishment module 504 is used to establish a communication group based on M normal processes; The communication module 505 is used to perform communication with M-1 normal processes through the many-to-many communication interface of the communication group. The M-1 normal processes include all processes other than the first process among the M normal processes.

[0168] In an optional embodiment, the apparatus further includes a size generation module 506. The size generation module 506 is used to generate a target size set; The communication module 505 is used to perform communication with M-1 normal processes based on the target size set through the many-to-many communication interface of the communication group; The target size set includes at least one of a first size set and a second size set. The first size set includes the number of first data allocated to each of the M-1 normal processes, and the second size set includes the number of data to be processed allocated to the first process by each of the M-1 normal processes.

[0169] In an optional embodiment, the target size set includes a first size set; the communication module 505 is used to divide the remaining first data into M-1 first data blocks according to the first size set; the remaining first data is first data other than the first data allocated to the first process; Through the many-to-many communication interface of the communication group, M-1 first data blocks are sent to M-1 normal processes; Among them, each of the M-1 first data blocks contains several first data items and corresponds to a normal process.

[0170] In an optional embodiment, the communication module 505 is further configured to receive M-1 second data blocks sent by M-1 normal processes through the many-to-many communication interface of the communication group. Integrity checks are performed on M-1 second data blocks using the first size set; Among them, each of the M-1 second data blocks contains several first calculation results. These several first calculation results are obtained by a normal process calling the expert network to perform calculations on the received several first data.

[0171] In an optional embodiment, the target size set includes a second size set; the communication module 505 is used to divide the calculation results belonging to M-1 normal processes output by the expert network of the first process into M-1 third data blocks according to the second size set; M-1 third data blocks are sent to M-1 normal processes through the many-to-many communication interface of the communication group; Among them, each of the M-1 third data blocks contains several second calculation results. The several second calculation results are obtained by the first process calling the local expert network to perform calculations on several data to be calculated in a fourth data block. A fourth data block contains several data to be calculated that are allocated to the first process by a normal process.

[0172] In an optional embodiment, the communication module 505 receives M-1 fourth data blocks sent by M-1 normal processes through the many-to-many communication interface of the communication group. Integrity checks are performed on M-1 fourth data blocks using the second size set; Among them, each of the M-1 fourth data blocks contains several data to be processed that were allocated to the first process by a normal process.

[0173] In an optional embodiment, the device further includes a parameter configuration module 507. The parameter configuration module 507 is used to configure communication parameters for the communication group based on the size of the communication group and the target size set. The communication parameters include the address of the first process's send buffer, the address of the receive buffer, the target size set, and the communication timeout.

[0174] In an optional embodiment, the apparatus further includes an aggregation module 508. The aggregation module 508 is used to aggregate the local computation result block with M-1 second data blocks to obtain a data aggregation result; The local calculation result block contains the calculation results of the first process calling the local expert network to perform calculations based on several allocated first data; each of the M-1 second data blocks is obtained by a normal process calling the expert network to perform calculations on several received first data.

[0175] In an optional embodiment, the communication group establishment module 504 is used to collect communication information of M normal processes; Based on the communication information, an initial communication group is created through the communication group creation interface; Each normal process in the initial communication group is reassigned a process identifier to obtain the communication group.

[0176] In an optional embodiment, the synchronization module 503 is further configured to synchronize the communication group information of the communication group to other normal processes; The communication group information includes at least one of the following: the original process identifier of each normal process in the communication group, the reassigned process identifier, and the size information of the communication group.

[0177] Figure 6This is a structural block diagram of a computer device according to an exemplary embodiment. Optionally, the computer device is a server. The computer device 600 includes a Central Processing Unit (CPU) 601, a system memory 604 including Random Access Memory (RAM) 602 and Read-Only Memory (ROM) 603, and a system bus 605 connecting the system memory 604 and the CPU 601. The computer device 600 also includes a basic input / output system (I / O system) 606 to facilitate information transfer between various devices within the computer device, and a mass storage device 607 for storing an operating system 613, application programs 614, and other program modules 615.

[0178] The basic input / output system 606 includes a display 608 for displaying information and an input device 609 for user input, such as a mouse or keyboard. Both the display 608 and the input device 609 are connected to the central processing unit 601 via an input / output controller 610 connected to the system bus 605. The basic input / output system 606 may also include the input / output controller 610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 610 also provides output to a display screen, printer, or other types of output devices.

[0179] The mass storage device 607 is connected to the central processing unit 601 via a mass storage controller (not shown) connected to the system bus 605. The mass storage device 607 and its associated computer device-readable media provide non-volatile storage for the computer device 600. That is, the mass storage device 607 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0180] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 604 and mass storage device 607 described above can be collectively referred to as memory.

[0181] According to various embodiments of this disclosure, the computer device 600 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 600 can be connected to the network 611 via a network interface unit 612 connected to the system bus 605, or it can use the network interface unit 612 to connect to other types of networks or remote computer device systems (not shown).

[0182] The memory also includes one or more programs stored in the memory, and the central processing unit 601 executes the one or more programs to implement all or part of the steps applied to the data processing method.

[0183] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the data processing method provided in the above method embodiments.

[0184] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method provided in the above-described method embodiments.

[0185] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0186] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0187] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method is executed by a first process among multiple processes. The data to be processed in the first process includes multiple first data sets. Each of the multiple processes is used to invoke an expert network to perform data processing. The multiple first data sets can be assigned to processes with probabilities greater than a probability threshold for processing. The method includes: Monitor the running status of the multiple processes; In response to the detection of a faulty process, the probability of the faulty process being assigned to the first data is adjusted to a value less than the probability threshold. The first data is used by the corresponding process to call the expert network for data processing. Generate a target size set; based on the target size set, perform communication with other normal processes besides the first process through the many-to-many communication interface of the communication group; The target size set is generated based on the routing allocation results of the plurality of first data, and is used to dynamically adjust the data block size corresponding to the other normal processes.

2. The method according to claim 1, characterized in that, The step of adjusting the probability of the faulty process being assigned to the first data to a value less than the probability threshold in response to detecting a faulty process includes: In response to the detection of the faulty process, an original probability set is calculated, the original probability set including the original probability of each of the plurality of first data being assigned to each of the plurality of processes; In the original probability set, the original probability corresponding to the faulty process is reduced to a penalty probability to obtain a modified probability set, wherein the penalty probability corresponding to the faulty process in each probability subset of the modified probability set is less than the probability threshold; wherein, a probability subset contains the probability of allocating a first data to each of the plurality of processes.

3. The method according to claim 2, characterized in that, In the original probability set, the original probability corresponding to the faulty process is reduced to a penalty probability to obtain a modified probability set, including: In the original probability set, the original probability corresponding to the faulty process is set to negative infinity to obtain the corrected probability set; or, In the original probability set, the original probability corresponding to the faulty process is multiplied by a penalty factor to obtain the corrected probability set, wherein the penalty factor is greater than 1 when the original probability corresponding to the faulty process is negative, and the penalty factor is less than 1 when the original probability corresponding to the faulty process is positive.

4. The method according to any one of claims 1 to 3, characterized in that, The monitoring of the running status of the multiple processes includes: The running status of the multiple processes is monitored through a heartbeat interaction mechanism.

5. The method according to claim 4, characterized in that, The monitoring of the running status of the multiple processes through the heartbeat interaction mechanism includes: For a target process, if no heartbeat packet is received from the target process within a preset time or if the information contained in the received heartbeat packet from the target process is incorrect, the target process is determined to be the faulty process. The target process is any process other than the first process among the plurality of processes.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: In response to the detection of the faulty process, the faulty process information of the faulty process is synchronized to the normal process via broadcast. The fault process information includes at least one of the following: the process identifier of the fault process, the fault type, and the fault occurrence time.

7. The method according to any one of claims 1 to 3, characterized in that, The first process is a temporary main process, and the number of normal processes among the plurality of processes is M, where M is a positive integer greater than one. The method further includes: The communication group is established based on M normal processes.

8. The method according to claim 7, characterized in that, The target size set includes at least one of a first size set and a second size set. The first size set includes the amount of first data allocated to each normal process in M-1 normal processes. The second size set includes the amount of data to be processed allocated to the first process by each normal process in the M-1 normal processes. The M-1 normal processes include processes other than the first process among the M normal processes.

9. The method according to claim 8, characterized in that, The target size set includes the first size set; The process of communicating with other normal processes besides the first process through the many-to-many communication interface of the communication group based on the target size set includes: According to the first size set, the remaining first data is divided into M-1 first data blocks; the remaining first data is the first data other than the first data allocated to the first process; The M-1 first data blocks are sent to the M-1 normal processes through the many-to-many communication interface of the communication group; Each of the M-1 first data blocks contains several first data items and corresponds to a normal process.

10. The method according to claim 9, characterized in that, The method further includes: The M-1 second data blocks sent by the M-1 normal processes are received through the many-to-many communication interface of the communication group. Integrity verification is performed on the M-1 second data blocks using the first size set; Each of the M-1 second data blocks contains several first calculation results, which are obtained by a normal process calling an expert network to perform calculations on several received first data.

11. The method according to claim 8, characterized in that, The target size set includes the second size set; The process of communicating with other normal processes besides the first process through the many-to-many communication interface of the communication group based on the target size set includes: According to the second size set, the calculation results belonging to the M-1 normal processes output by the expert network of the first process are divided into M-1 third data blocks; The M-1 third data blocks are sent to the M-1 normal processes through the many-to-many communication interface of the communication group; Among them, each of the M-1 third data blocks contains several second calculation results, which are obtained by the first process calling the local expert network to perform calculations on several data to be calculated in a fourth data block; a fourth data block contains several data to be calculated allocated to the first process by a normal process.

12. The method according to claim 11, characterized in that, The method further includes: Through the many-to-many communication interface of the communication group, receive M-1 fourth data blocks sent by the M-1 normal processes; Integrity verification is performed on the M-1 fourth data blocks using the second size set; Each of the M-1 fourth data blocks contains several pieces of data to be processed, which are allocated to the first process by a normal process.

13. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the size of the communication group and the target size set, configure communication parameters for the communication group; The communication parameters include the address of the sending buffer of the first process, the address of the receiving buffer, the target size set, and the communication timeout.

14. The method according to claim 7, characterized in that, The method further includes: The local computation result block is aggregated with M-1 second data blocks to obtain the data aggregation result; The local calculation result block contains the calculation results of the first process calling the local expert network to perform calculations based on a number of allocated first data; each of the M-1 second data blocks is obtained by a normal process calling the expert network to perform calculations on a number of received first data.

15. The method according to claim 7, characterized in that, The establishment of the communication group based on M normal processes includes: Collect communication information from the M normal processes; Based on the communication information, an initial communication group is created through the communication group creation interface; The communication group is obtained by reassigning process identifiers to each normal process in the initial communication group.

16. The method according to claim 7, characterized in that, The method further includes: Synchronize the communication group information of the aforementioned communication group to other normal processes; The communication group information includes at least one of the following: the original process identifier of each normal process in the communication group, the reassigned process identifier, and the size information of the communication group.

17. A data processing device, characterized in that, The device operates a first process among multiple processes. The data to be processed in the first process includes multiple first data. Each of the multiple processes is used to call an expert network to perform data processing. The multiple first data can be assigned to processes with probabilities greater than a probability threshold for processing. The device includes: A monitoring module is used to monitor the running status of the multiple processes; An adjustment module is used to adjust the probability of the faulty process being assigned to the first data to a value less than the probability threshold in response to the detection of a faulty process. The first data is used for the corresponding process to call the expert network for data processing. The size generation module is used to generate a set of target sizes; The communication module is used to perform communication with other normal processes besides the first process through the many-to-many communication interface of the communication group, based on the target size set. The target size set is generated based on the routing allocation results of the plurality of first data, and is used to dynamically adjust the data block size corresponding to the other normal processes.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the data processing method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 16.

Images