Data processing method and question answering method based on acceleration equipment

Abstract

The invention discloses a data processing method and a question answering method based on acceleration equipment, and relates to the technical field of data processing, under the condition that an equilibrium condition is not met, a to-be-scheduled expert module is determined through a first actual load capacity of each target expert module, an expert module copy is set for the to-be-scheduled expert module, and the to-be-scheduled expert module is determined; the purpose of setting the copy is to ensure that at least one complete expert module type is reserved in all target acceleration devices at any moment, the interruption condition of subsequent reactivation caused by missing of a certain expert module type at any moment is avoided, and the deployment continuity is ensured. The problems that the continuity of service deployment is affected and the reasoning efficiency is reduced due to interruption processing during scheduling updating are solved, an unbalanced mechanism is determined in time through satisfaction detection of equilibrium conditions, and all target acceleration devices keep original expert module types at any time to complete scheduling processing without interruption, so that the scheduling efficiency is improved. And the technical effects of scheduling processing efficiency and load balancing are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and question-answering method based on an acceleration device. Background Technology

[0002] During distributed deployment, the expert modules of the hybrid expert system of the pre-trained model are statically allocated to each acceleration device. When the number of tokens is large and the load is unbalanced, the scheduling of expert modules on each acceleration device is adjusted. The expert modules with the corresponding load under the overload condition of a specific acceleration device are adjusted to other acceleration devices with light load. Specifically, the expert modules with a large load on a specific acceleration device are first unloaded and added to other acceleration devices with light load before being activated. This interruption method maintains the scheduling balance. However, the interruption process will affect the continuity of the deployment service and further reduce the inference efficiency.

[0003] Therefore, how to avoid interruptions while ensuring balanced deployment and scheduling of expert modules across acceleration devices to improve inference efficiency is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a data processing method and a question-answering method based on an acceleration device to at least solve the problem in related technologies where interruptions during scheduling processing affect the continuity of deployed services and reduce inference efficiency.

[0005] This application provides a data processing method based on an acceleration device, including: Receive the current data request and determine the target expert module that corresponds to the current data request and is deployed on the target acceleration device; Obtain the first actual load of each of the target expert modules and the second actual load of each of the target acceleration devices; Determine whether the target acceleration device meets the balance condition based on each of the second actual loads; If the equilibrium condition is not met, then the expert module to be scheduled is determined based on the first actual load of each of the target expert modules, and a corresponding expert module copy is set for the expert module to be scheduled, so that the target acceleration device that does not meet the equilibrium condition can reach the equilibrium condition.

[0006] This application also provides a question-answering method based on a hybrid expert system, including: Obtain the question-and-answer dataset corresponding to the reasoning question; The question-and-answer dataset is input into a hybrid expert system to output response information; The hybrid expert system is obtained by reasoning and optimization through the steps of the data processing method based on the acceleration device described above.

[0007] This application addresses the problem that conventional scheduling updates require interruption processing for load balancing, which disrupts service continuity and reduces inference efficiency. First, it identifies the target expert module deployed on the target acceleration device corresponding to the current data request, thus determining the target acceleration device and expert module involved in scheduling for the current data request. Second, it determines whether each target acceleration device meets load balancing conditions based on its second actual load. Using the load on the acceleration device as a load balancing standard allows for timely detection of resource mismatches, effective load balancing, reduced task processing latency, and improved business efficiency. Third, if load balancing conditions are not met, the application determines the expert module to be scheduled based on the first actual load of each target expert module. This can be two expert modules on different acceleration devices that need to share the full load, or an expert module on one acceleration device that needs to share a portion of the load with other acceleration devices. This enhances the flexibility and diversity of the scheduling solution. Finally, a corresponding expert module replica is set for the expert module to be scheduled. The purpose of this replication is to ensure that at any given time, all target acceleration devices retain at least one complete set of expert module types, preventing interruptions such as the loss of an expert module type leading to subsequent reactivation, thus ensuring deployment continuity. In the first scheduling method, this involves adding the expert replica of the expert module to be scheduled to both acceleration devices, ensuring uninterrupted deployment. In the second scheduling method, if there were no expert replicas initially, a portion of the load of the expert module to be scheduled is distributed to the expert replicas. This is done by adding the expert replica to other acceleration devices, thus mitigating the overload on the acceleration device hosting the expert module. Alternatively, if expert replicas already exist, some may need to be released before setting a corresponding expert replica for the expert module to be scheduled, achieving load balancing. In short, setting corresponding expert module replicas for the expert module to be scheduled ensures that the load of the expert module is distributed across the corresponding acceleration devices through replication, achieving load balancing across all acceleration devices.

[0008] Therefore, this solution addresses the problem that conventional solutions require interruption processing to achieve load balancing during scheduling updates, which affects the continuity of deployment services and reduces inference efficiency. It achieves this by promptly identifying imbalances through load balancing condition checks, determining the corresponding expert module replicas based on the first actual load of each target expert module, and ensuring that all target acceleration devices retain their original expert module types at any given time to complete scheduling without interruption. This improves scheduling efficiency and load balancing performance. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a data processing method based on an acceleration device provided in this application embodiment; Figure 2 This embodiment provides a schematic diagram of a temporary borrowing of an expert module copy. Figure 3 A schematic diagram illustrating the addition of a copy of an expert module, provided as an embodiment of this application; Figure 4 This is a schematic diagram illustrating the addition of an expert module after it has been released, provided that the module has an expert copy, as an embodiment of this application. Figure 5 A schematic diagram illustrating the establishment of a second logical mapping relationship provided in an embodiment of this application; Figure 6 This is a structural diagram of a data processing device based on an acceleration device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] The specific application environment architecture or specific hardware architecture on which the execution of data processing methods and question-answering methods based on accelerated devices depends is described here.

[0015] In the parallel inference process of a hybrid expert system, different expert modules are distributed across different acceleration devices. Each acceleration device stores and runs specific expert modules. For example, in a Graphics Processing Unit (GPU) cluster, 128 expert modules are evenly distributed across 4 GPUs, with each GPU responsible for the computation of 32 expert modules. The conventional approach is to statically allocate expert modules to different acceleration devices, leading to uneven call frequencies. Concentrated calls to popular expert modules can overload specific acceleration devices, leaving others idle, resulting in low resource utilization. During scheduling, expert modules with high loads on specific acceleration devices are unloaded and added to other, less loaded acceleration devices for activation. However, during unloading and activation, some acceleration devices may not have the unloaded expert modules, leading to missing modules. Furthermore, the unloading and activation process requires system downtime to replace expert modules before activation can proceed, causing interruptions, affecting the continuity of server deployment, and reducing inference efficiency. The data processing method based on acceleration devices provided in this application solves the above technical problems.

[0016] Figure 1 A flowchart of a data processing method based on an acceleration device provided in this application embodiment is shown below. Figure 1 As shown, the method includes: S11: Receive the current data request and determine the target expert module that corresponds to the current data request and is deployed on the target acceleration device; S12: Obtain the first actual load of each target expert module and the second actual load of each target acceleration device; S13: Determine whether the target acceleration device meets the balancing condition based on the actual load of each second device; if it does not meet the condition, proceed to step S14; if it does meet the condition, proceed to step S15. S14: Determine the expert module to be scheduled based on the first actual load of each target expert module, and set the corresponding expert module copy for the expert module to be scheduled so that the target acceleration device that does not meet the balancing condition can reach the balancing condition; and obtain the next data as a new current data request, and return to step S11; S15: Obtain the next data request as the new current data request, and return to step S11.

[0017] Specifically, the system receives the current data request and, based on the corresponding terms in the parallel inference process of the hybrid expert system, determines the corresponding target expert module. This target expert module is deployed on the target acceleration device. Here, the target expert module is the expert module that participates in the computation within the terms. It should be noted that the current data request is a scheduling request for historical load data within the load balancing scheduling mechanism in this embodiment, distinct from requests during the model's parallel inference process.

[0018] In step S12, the first actual load of each target expert module and the second actual load of each target acceleration device are obtained. It should be noted that the first actual load and the second actual load correspond to two dimensions. The first actual load is the number of calls for each target expert module itself, and the second actual load is the sum of the first actual loads of the deployed target expert modules for each target acceleration device.

[0019] In step S13, the balance condition of the target acceleration devices is determined based on the second actual load of each target acceleration device. This balance condition is not limited to a single specific target acceleration device; it is determined by globally checking all target acceleration devices. The core of balance is that the resource utilization of acceleration devices tends to converge, with no significant overload or limitation. This can be determined by the difference between each second actual load and the balance degree. If the difference is large, it indicates that all target acceleration devices as a whole do not meet the balance condition. Alternatively, the second actual loads can be compared; if the difference is greater than a preset value, the balance condition is not met. Alternatively, settings for computational load, storage load, and communication load can be used. For example, whether the utilization difference of each acceleration device meets a first preset value, and whether the standard deviation of computation time meets a second preset value, can determine whether the computational load meets the balance condition; whether the difference in memory usage of each acceleration device meets a third preset value, and whether the memory overflow or idle rate of any single acceleration device exceeds a fourth preset value, can determine whether the storage load meets the balance condition; and whether the difference in data transmission latency between acceleration devices is less than or equal to a fifth preset value, can determine whether the communication load meets the balance condition. The specific equilibrium conditions can be determined by comparison based on the actual situation.

[0020] If preset conditions are not met, the expert modules to be scheduled are determined based on the first actual load of each target expert module. It should be noted that the expert module to be scheduled is the one whose scheduling needs to be changed during the scheduling process, causing an imbalance in the second actual load of the corresponding acceleration device. There can be one or more expert modules to be scheduled in each scheduling cycle; this is not limited here. The scheduling process can involve swapping two expert modules on their respective deployment acceleration devices during each scheduling cycle, or it can involve distributing their first actual load on another acceleration device during each scheduling cycle; this is not limited here. The determination of the expert module to be scheduled can be based solely on the first actual load of the target expert module, or it can be based on a combination of the second actual load of the target expert module deployed on the target acceleration device; this is not limited here.

[0021] The expert module to be scheduled is determined based on the first actual load of the target expert module. In the current scheduling process, the target expert module corresponding to the largest first actual load is determined as the expert module to be scheduled. Alternatively, considering that the current target expert module already has a copy in the previous scheduling process, but the load of the target acceleration device is currently unbalanced and needs to be released accordingly, this scheduling process will not be among the target expert modules that already have a copy. Instead, it needs to find the target expert module with the current first actual load that does not have a copy and determine it as the expert module to be scheduled.

[0022] The expert modules to be scheduled are determined based on the first actual load of the target expert module and the second actual load of the target acceleration device. Two expert modules are selected for scheduling to facilitate subsequent mutual scheduling. First, the second actual load of the target acceleration device is compared. Only if the difference between the average of the maximum and minimum second actual loads and the globally determined average meets a threshold can the corresponding target acceleration device be determined to meet the scheduling conditions. Then, among the acceleration devices with the maximum second actual load, the target expert module with the maximum first actual load is selected as one of the scheduling expert modules. For the target acceleration devices with the minimum second actual load, another scheduling expert module is selected, and mutual scheduling is then performed.

[0023] For each expert module to be scheduled, a corresponding expert module copy is set. This expert module copy is an additional copy set. If there is only one expert module in all acceleration devices, the initial expert module copy is 1. It should be noted that there are three ways to set copies: First, temporarily borrow an expert module copy, interleaving the positions of two scheduled expert modules, with the expert module copy representing the full load of that scheduled expert module. Second, rely on the establishment of expert module copies on other acceleration devices, using these expert module copies to distribute the load of the expert module to be scheduled to other acceleration devices. Third, if an expert module copy already exists from a previous scheduling process (i.e., an initial copy exists), release that copy to add other expert modules without allocated expert module copies, assigning expert module copies to the positions where the original copies were released. These three copy setting methods are parallel schemes; any one can be chosen arbitrarily in actual use. The first two copy setting methods correspond to global scheduling, while the third copy setting method corresponds to local scheduling.

[0024] Taking acceleration device A and acceleration device B as examples, Figure 2 This embodiment provides a schematic diagram of a temporary borrowing of an expert module copy, as shown below. Figure 2As shown, expert module A2 on acceleration device A has a load of 400 and is designated as the first expert module to be scheduled. Expert module A1 on acceleration device B has a load of 200 and is designated as the second expert module to be scheduled. The final scheduling of the two expert modules involves interleaving their positions and adding them to copies of their respective expert modules. These copies are indicated by dashed boxes on the acceleration devices; that is, acceleration device A adds a copy of expert module A1 (expert module A1'), and acceleration device B adds a copy of expert module A2 (expert module A2'). The load corresponding to each expert module is then added to its respective copy. After scheduling is complete, the expert modules are removed from their original positions on the acceleration devices. The entire process retains the types of all expert modules at any given time without requiring interruption.

[0025] Figure 3 This application provides a schematic diagram of adding a copy of an expert module, as shown in the embodiment of the present application. Figure 3 As shown, the expert module does not have a replica. By adding an expert replica, the load of expert module B0 (expert module B0) on acceleration device C is 400. As an expert module to be scheduled, it needs to transfer part of its load to acceleration device B. Therefore, a replica of expert module B0 (expert module B0') is created on acceleration device B to achieve load balancing.

[0026] Figure 4 This application provides an embodiment of an example of adding an expert module after it has been released, provided that the expert module has an expert copy. Figure 4 As shown, based on the current data request, if the deployment is carried out according to the original expert replica situation, the load of the corresponding expert module is uneven across the acceleration devices. Therefore, it is necessary to release the replica (expert module C1') corresponding to expert module C1, add the replica of expert module C2 to the position of the already released replica of expert module C1, and transfer part of the load of expert module C2 to the replica (expert module C2') of expert module C2, so that the load corresponding to the current data request is balanced.

[0027] If the load balancing condition is met, after deployment, no scheduling is required. Instead, the next data request is used as the new current data request for decision-making and deployment.

[0028] This application addresses the problem that conventional scheduling updates require interruption processing to achieve load balancing, impacting service continuity and reducing inference efficiency. First, it identifies the target expert module deployed on the target acceleration device corresponding to the received current data request. This identifies the target acceleration device and target expert module involved in scheduling processing under the current data request. Second, it determines whether each target acceleration device meets load balancing conditions based on the second actual load of the target acceleration device. Using the load on the acceleration device as a load balancing standard allows for timely detection of resource mismatches, effective load balancing, reduced task processing latency, and improved business efficiency. Third, if load balancing conditions are not met, the first actual load of each target expert module determines the expert module to be scheduled. This can be two expert modules on different acceleration devices that need to mutually schedule the full load, or an expert module on one acceleration device that needs to schedule a portion of the load from other acceleration devices. This achieves flexibility and diversity in the scheduling processing scheme. Finally, a corresponding expert module replica is set for the expert module to be scheduled. The purpose of this replication is to ensure that at any given time, all target acceleration devices retain at least one complete set of expert module types, preventing interruptions such as the loss of an expert module type leading to subsequent reactivation, thus ensuring deployment continuity. In the first scheduling method, this involves adding the expert replica of the expert module to be scheduled to both acceleration devices, ensuring uninterrupted deployment. In the second scheduling method, if there were no expert replicas initially, a portion of the load of the expert module to be scheduled is distributed to the expert replicas. This is done by adding the expert replica to other acceleration devices, thus mitigating the overload on the acceleration device hosting the expert module. Alternatively, if expert replicas already exist, some may need to be released before setting a corresponding expert replica for the expert module to be scheduled, achieving load balancing. In short, setting corresponding expert module replicas for the expert module to be scheduled ensures that the load of the expert module is distributed across the corresponding acceleration devices through replication, achieving load balancing across all acceleration devices.

[0029] In some embodiments, determining whether the target acceleration device meets the balancing condition based on the second actual load of each unit includes: In the current scheduling process, the average second actual load is determined based on the current second actual load of each target acceleration device. Determine the current maximum second actual load within each current second actual load; The current balance is determined based on the current maximum second actual load and the average second actual load. If the current equilibrium exceeds the preset equilibrium, it is determined that each target acceleration device does not meet the equilibrium condition.

[0030] Specifically, during each scheduling process, the average second actual load needs to be determined based on the current second actual load of each target acceleration device. The current balance is obtained by subtracting the current maximum second actual load from the average second actual load. If the current balance exceeds the preset balance, it is determined that each target acceleration device does not meet the balance condition. The smaller the specific value of the current balance, the better. Theoretically, the minimum is 1, indicating that the current maximum second actual load is getting closer to the average second actual load.

[0031] The balance degree provided in this embodiment, determined based on the actual load of each second load, uses the relationship between the balance degree and the preset balance degree to determine whether the balance condition is met. This avoids system crashes caused by single-point overload, focuses on overall balance to avoid the one-sidedness of local comparison, directly reflects the deviation of all acceleration devices from the overall level, and covers the global state of the system.

[0032] In some embodiments, determining the expert module to be scheduled based on the first actual load of each target expert module includes: Get the preset number of scheduling attempts; In the current scheduling count, the second actual load of each target acceleration device is sorted in descending order to obtain the first sequence; The first average load is determined based on the current maximum second actual load and the current minimum second actual load in the first sequence; If the difference between the first average load and the average second actual load is less than or equal to the first threshold, then the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load are determined to meet the scheduling conditions. The target expert module corresponding to the current maximum first actual load in the target acceleration device corresponding to the current maximum second actual load is taken as the current first expert module to be scheduled. The reference load is determined based on the first average load, the current maximum first actual load of the first expert module to be scheduled, and the current maximum second actual load. Within the first actual load of each target expert module in the target acceleration device corresponding to the current minimum second actual load, select the first actual load with the smallest difference from the reference load as the target first actual load; take the target expert module corresponding to the target first actual load as the current second expert module to be scheduled; record the current scheduling count; After determining the target acceleration devices after scheduling, return to the step of sorting the second actual load of each target acceleration device in descending order to obtain the first sequence, obtain the new first sequence to determine the expert module to be scheduled, and so on, until the current scheduling number reaches the preset scheduling number, and complete the process of determining the expert module to be scheduled.

[0033] Specifically, the process of determining the expert module to be scheduled is a global scheduling process, with each acceleration device processing in parallel. Scheduling is performed at least once, therefore, a preset number of scheduling attempts is required to ensure scheduling balance. In the current number of scheduling attempts, the current maximum second actual load (maxP) and the current minimum second actual load (minP) of each target acceleration device determine the first average load (P2). If the difference between the first average load (P2) and the average second actual load (P1) is less than or equal to a first threshold, it indicates that the expert modules on the target acceleration devices corresponding to the current maximum and minimum second actual loads can be used as expert modules to be scheduled, meaning that the target acceleration devices corresponding to the current maximum and minimum second actual loads meet the scheduling conditions.

[0034] The target expert module corresponding to the current maximum first actual load of the target acceleration device corresponding to the current maximum second actual load is taken as the current first expert module to be scheduled.

[0035] In the target expert modules of the target acceleration device corresponding to the current minimum second actual load, it is necessary to determine the current second expert module to be scheduled. At this time, it is necessary to select the first actual load with the smallest distance from the reference load value among the first actual loads of each target expert module in the target acceleration device corresponding to the current minimum second actual load as the target first actual load, and the target expert module corresponding to it as the current second expert module to be scheduled.

[0036] It should be noted that the process of determining the reference load is obtained by using the first average load, the current maximum first actual load of the current first expert module to be scheduled, and the current maximum second actual load. Specifically, the first average load is added to the current maximum first actual load of the current first expert module to be scheduled to obtain a first value; the reference load is determined by subtracting the current maximum second actual load from the first value.

[0037] It is worth noting that the above embodiments only record the two expert modules to be scheduled during the current scheduling process. After the scheduling is completed, further scheduling is required. Therefore, it is necessary to record the current number of scheduling attempts until the preset number of scheduling attempts is reached, at which point the scheduling ends. Each scheduling process determines the expert modules to be scheduled according to the determination process described in the above embodiments.

[0038] In addition to limiting the number of scheduling attempts to a preset limit, another method can be used to determine whether the balancing condition is met by calculating the balance degree corresponding to the current second actual load of each target acceleration device after each scheduling is completed, and checking if it exceeds the preset balance degree. If it is met, no further scheduling is required.

[0039] This embodiment, during each scheduling process, identifies two expert modules to be scheduled, facilitating the staggered transfer of load between them. The target acceleration device with the smallest actual load is locked; this device is already under low load and has sufficient computing power / memory to handle the new expert module without overloading itself. Then, the target expert module on this device with the smallest actual load relative to the reference load is selected, ensuring that the expert module itself has low load pressure and its migration has minimal impact on the original device's load. This maximizes overall load balance, avoids secondary imbalances, and avoids blindly selecting the expert module with the smallest load; instead, it focuses on minimizing the difference, resulting in a smoother overall load distribution after scheduling. It considers both the overall load at the device level and the individual load of expert modules, preventing load skew between devices caused by the migration of a single expert.

[0040] In some embodiments, setting up a corresponding expert module copy for the expert module to be scheduled includes: In the current scheduling count, the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load that meet the scheduling conditions are respectively designated as the current first acceleration device and the current second acceleration device. Establish temporary expert module copy locations on the current first acceleration device and the current second acceleration device; At the location of the temporary expert module copy on the current second acceleration device, set the corresponding first expert module copy for the current first expert module to be scheduled, so as to process the first actual load corresponding to the current first expert module to be scheduled to the first expert module copy in full, and delete the expert module location corresponding to the current first expert module to be scheduled on the current first acceleration device. At the location of the temporary expert module copy on the current first acceleration device, set the corresponding second expert module copy for the current second expert module to be scheduled, so as to process the first actual load corresponding to the current second expert module to be scheduled in full to the second expert module copy, and delete the expert module location corresponding to the current second expert module to be scheduled on the current second acceleration device. Increment the current scheduling count by 1 to obtain the new current scheduling count, and return to the step of setting the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load that meet the scheduling conditions as the current first acceleration device and the current second acceleration device, respectively. Continue in this manner until the new current scheduling count reaches the preset scheduling count, and complete the process of setting up the expert module replica of the expert module to be scheduled.

[0041] Specifically, each target expert module here has only an initial expert module copy; that is, there is only one target expert module of each type. Temporary expert module copy positions are those temporarily added as additional expert module copies. Considering that the entire scheduling process involves multiple acceleration devices, and multiple scheduling attempts are needed to fully schedule the load at the device level, in each scheduling process, the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load that meet the scheduling conditions are respectively designated as the current first acceleration device and the current second acceleration device. Temporary expert module copy positions are then established on the current first acceleration device and the current second acceleration device, such as... Figure 2 As shown, the first actual load corresponding to the current first expert module to be scheduled (expert module A2) is fully processed and transferred to the first expert module copy, and the expert module position corresponding to the current first expert module to be scheduled on the current first acceleration device is deleted. The first actual load corresponding to the current second expert module to be scheduled (expert module A1) is fully processed and transferred to the second expert module copy, and the expert module position corresponding to the current second expert module to be scheduled on the current second acceleration device is deleted.

[0042] After completing the current scheduling, record the current scheduling count, and continue with new scheduling until the new current scheduling count reaches the preset scheduling count. This is the process of setting up the expert module replica.

[0043] The provision in this embodiment, which borrows temporary expert module copies, allows two modules to be scheduled in each scheduling process to be cross-scheduled through temporary expert module copies. This ensures that the complete set of expert module types is retained at any time, avoiding interruptions, guaranteeing the continuity of deployment services and automated deployment processing, and improving the inference efficiency of inference tasks.

[0044] In other embodiments, the expert modules to be scheduled are determined based on the first actual load of each target expert module, including: Get the number of pre-reserved expert module copy locations; The second sequence is obtained by sorting the first actual load of each target expert module in descending order. In the second sequence, the first target expert module corresponding to the current maximum first actual load is taken as the expert module to be scheduled, so as to perform balancing processing on the expert module to be scheduled, to obtain the first sub-actual load, and to subtract 1 from the number of expert module replica positions to obtain the new number of expert module replicas. The first sub-actual load and the first actual load corresponding to other target expert modules (excluding the first target expert module) are sorted in descending order to obtain a new second sequence. Then, the first target expert module corresponding to the current largest first actual load is selected as the expert module to be scheduled in the second sequence to obtain a new expert module to be scheduled. This process is repeated until the number of new expert module replicas is 0 to obtain the corresponding expert module to be scheduled.

[0045] Specifically, considering that the original target expert modules have no additional expert replicas, the number of expert module replica positions is pre-reserved during this process. A second sequence is obtained by descending the order of the first actual load of each target expert module. The first target expert module corresponding to its current maximum first actual load is selected as the expert module to be scheduled. These expert modules are then balanced, meaning their loads are evenly distributed to obtain the first sub-actual load. If the load of a target expert module is not divisible by 2, the difference between the balanced loads is 1.

[0046] Since the number of expert module replica positions is fixed and will be used up, it is necessary to subtract 1 from the number of expert module replica positions after completing one leveling process to obtain the new number of expert module replicas.

[0047] After completing the current scheduling, the actual load of each first sub-sub and the actual load of the other target expert modules (excluding the first target expert module) are sorted in descending order to obtain a new second sequence. New expert modules to be scheduled are then found until the number of new expert module replicas is 0, so as to obtain the corresponding expert modules to be scheduled.

[0048] In this embodiment, a scheduled expert module is determined in each scheduling process. When there is no expert copy in the original expert module, the first target expert module corresponding to the current maximum first actual load is used as the scheduled expert module, which simplifies the determination process. The target expert module with the current maximum first actual load is used as the scheduled expert module. Prioritizing scheduling can directly reduce the system's maximum load and improve the overall inference efficiency.

[0049] In some embodiments, when there are no expert module copies for the other non-scheduled expert modules of the expert module to be scheduled, a corresponding expert module copy is set for the expert module to be scheduled, including: The third sequence is obtained by sorting the first sub-actual load of each expert module to be scheduled in descending order. Obtain the third acceleration device corresponding to the current minimum second actual load; At the expert module replica position on the third acceleration device, set the corresponding first expert module replica for the expert module to be scheduled in the third sequence that corresponds to the current maximum first sub-actual load, so as to move the first sub-actual load to the first expert module replica, and decrement the number of expert module replica positions by 1 to obtain the new number of expert module replicas. In the third sequence, delete the first sub-actual load of the expert module to be scheduled corresponding to the first expert module replica, obtain a new third sequence, and return to the step of obtaining the third acceleration device corresponding to the current minimum second actual load, so as to obtain a new third acceleration device. This process continues until the number of new expert module replica positions is 0, thus completing the expert module replica setting of the expert module to be scheduled.

[0050] Specifically, when there are no expert module replicas among the other non-scheduled expert modules of the expert module to be scheduled (i.e., only the initial expert module replica exists), it should be noted that the initial expert module replica is only 1, and any additional expert module replicas added are extra expert module replicas. A third sequence needs to be obtained by descending the first sub-actual load of each expert module to be scheduled for the current scheduling count. The expert module replica position is set in the acceleration device corresponding to the current minimum second actual load. The expert module to be scheduled corresponding to its current maximum first sub-actual load is set as the corresponding first expert module replica, its first sub-actual load is transferred to the first expert module replica, and the number of expert module replica positions is reduced by 1 to obtain the new number of expert module replicas.

[0051] Furthermore, in the third sequence, the scheduled tasks that have been completed are deleted, and the first sub-actual load of the expert module to be scheduled corresponding to the first expert module replica is set to obtain a new third sequence, until the number of new expert module replica positions is exhausted.

[0052] It should be noted that when constructing the initial allocation strategy for expert modules, allocation is performed globally. Assume the model performs inference services on G cards (i.e., there are a total of G computing accelerator cards), and let the number of expert modules be E. Each computing accelerator card holds R copies of expert modules (total number of expert module copies T = G × R, T ≥ E). If T = E, then each expert module has only one copy, with no additional copies. It's important to clarify that this single copy for each expert module includes the original copy that already exists, not any newly added copies. If T > E, then there are F = TE additional copies, which can belong to one or more expert modules. The steps are as follows: 1.1 Based on the load data of the routing experts, determine the number of replicas for each expert. If T=E, then one replica is needed for each expert. If T>E, then the following 1.2 allocation is required.

[0053] 1.2 If each expert is assigned a position, there will still be F replicas to be assigned. Let the number of replicas for each expert be N = [n0, n1, ..., nE−1], where ni represents the number of replicas for expert module i. Also, assume that the number of replicas for each expert module does not exceed m. Assume that the historical load data for each expert module is D = [d0, d1, ..., dE−1], where di represents the load of expert module i.

[0054] If there are still replicas to be allocated, select the expert module mi with the highest load rate. If the number of replicas of i is less than m, add one replica to that expert module, updating it to nmi = nmi + 1, where the load rate is determined by pi = di / ni, and decrease the number of remaining replicas by 1. If the number of replicas of the expert module i with the highest load rate is equal to m, remove expert module i from the sorting list, select the expert module j with the highest load rate among the remaining expert modules, and if the number of replicas of j is less than m, add one replica to that expert module, updating it to nj = nj + 1, and decrease the number of remaining replicas by 1. Repeat the above process until there are no replicas to be allocated.

[0055] 1.3 Based on the number of expert module replicas obtained in steps 1.1 and 1.2, the expert module allocation strategy on each computing accelerator card is as follows: 1.3.1 Sort all expert module replicas in descending order of load rate. When an expert has multiple replicas (i.e., ni > 1), they have the same load rate, totaling T replicas, and wait for an initialization queue Q. 1.3.2 Considering the grouping structure of physical devices, initialize the number of available replicas on all computing accelerator cards to S=[R]*G, that is, each card contains R replica positions, and there are a total of G cards; initialize the sum of the load rates on each card to P=[0]*G; 1.3.3 Assign the first expert module replica in queue Q (i.e., the expert with the highest unassigned load rate) to the card with the lowest sum of current load rates and a non-zero number of available replicas; update the Q, S, and P values, remove the expert module replica from queue Q, decrease the number of available replica experts on the card by 1, and add the load rate corresponding to the aforementioned expert module replica to the sum of the card's load rates; repeat the above process until the number of available replicas on each card is zero, or Q is null. At this point, the initial allocation of expert modules on each card is complete. Meanwhile, based on the sum of the load rates on each card, P, the current load balancing degree max(P) / avg(P) can be calculated. The smaller the value, the better, with a theoretical minimum of 1.

[0056] For example: In step 1.2, expert modules 0, 1, 2, and 3 correspond to load quantities of 400, 300, 200, and 200 respectively, with a preset additional expert module replica of 2. Here, the expert module with the highest load is selected, and the first division by the number of replicas (1) is 400, which is still the highest load. At this time, the number of replicas of expert module 0 is increased by 1 to get 2, which means that expert module 0 is set to two. At this time, the load allocated to each expert module 0 is 200. At this time, there is still one preset additional replica left for expert modules 0, 1, 2, 3, and 0. At this time, expert module 1 has the highest load of 300, and its number of replicas is increased by 1 to get 2, which means that expert module 1 is set to two. At this time, the load allocated to each expert module 1 is 150. At this time, the preset additional replica quantity has been distributed, and it is now known which expert modules the additional expert replicas are specifically distributed to.

[0057] Step 1.3 determines which GPU card to place the identified expert modules and additional expert copies on. Following step 1.2, the numbers 0, 0, 1, 1, 2, 3 correspond to loads of 200, 200, 150, 150, 200, 200 respectively. These are arranged in descending order, with the initial queue Q being 200, 200, 200, 200, 150, 150 – a total of 6 experts. However, with two accelerator cards, and since the load is roughly evenly distributed, each accelerator card hosts 3 expert modules.

[0058] Step 1.3.3: Place the highest value of the expert load onto the accelerator card with the lowest load. Each time it is placed, the latest Q queue and the load status of each accelerator card will be counted.

[0059] In this embodiment, during the current scheduling, an expert module replica position is set in the acceleration device corresponding to the current minimum second actual load, so as to transfer the current maximum first sub-actual load by adding the replica position, thereby avoiding overload of the target acceleration device where the expert module to be scheduled was originally located.

[0060] In some embodiments, determining the expert module to be scheduled based on the first actual load of each target expert module includes: Obtain the first target number of expert module copy locations to be released; wherein, the first target number is less than or equal to the number of existing expert module copies for each target expert module; Obtain the second target expert module corresponding to the number of expert module copies that do not exist in each target expert module; The fourth sequence is obtained by sorting the first actual load of each second objective expert module in descending order. In the fourth sequence, the second target expert module corresponding to the current maximum first actual load is taken as the expert module to be scheduled, so as to perform balancing processing on the expert module to be scheduled, to obtain the second sub-actual load, and to subtract 1 from the first target quantity to obtain the new first target quantity. The actual load of each second sub-sub and the first actual load corresponding to the other second target expert modules in the second target expert module (excluding the expert module to be scheduled) are sorted in descending order to obtain a new fourth sequence. Then, the process is returned to the step of selecting the second target expert module corresponding to the current largest first actual load in the fourth sequence as the expert module to be scheduled, and so on, until the number of new first targets is 0, so as to obtain the corresponding expert modules to be scheduled.

[0061] Specifically, the first target number of expert module copies to be released is obtained. This number can be the original number of expert module copies of all expert modules, or it may be less than, or it may not need to be released. Therefore, the first target number is less than or equal to the number of existing expert module copies of each target expert module.

[0062] It should be noted that a second target expert module needs to be allocated for which there are no expert module copies in the current target expert module. The original expert module positions will be retained, only the copy positions will be released.

[0063] The first actual load of each second target expert module is sorted in descending order to obtain a fourth sequence. The second target expert module corresponding to the current largest first actual load is selected as the expert module to be scheduled and subjected to load balancing. The load balancing process is the same as in the above embodiment and is not limited here. The second sub-actual load is obtained, and the first target quantity is reduced by 1 to obtain a new first target quantity. If there is still a first target quantity, the expert modules to be scheduled are determined.

[0064] This embodiment provides a method to determine the number of expert module copies to be added based on the location of the expert module copy to be released when expert copies already exist. This method identifies the expert modules to be scheduled, thereby achieving local updates within the smallest possible scope, reducing the number of scheduling operations, and improving scheduling efficiency.

[0065] In some embodiments, when there are expert module copies for other non-scheduled expert modules of the expert module to be scheduled, a corresponding expert module copy is set for the expert module to be scheduled, including: Pre-select the target acceleration devices corresponding to the locations of the expert module copies to be released as the fourth acceleration devices; Find the fifth acceleration device corresponding to the current minimum second actual load among all the fourth acceleration devices; The fifth sequence is obtained by sorting the second sub-actual load of each expert module to be scheduled in descending order. Release the expert module copy on the fifth acceleration device, set the corresponding second expert module copy for the expert module to be scheduled corresponding to the current maximum second sub-actual load in the fifth sequence, and insert it into the released expert module copy position to move the second sub-actual load to the second expert module copy, and decrement the first target quantity by 1 to obtain the new first target quantity. In the fifth sequence, delete the second sub-actual load of the expert module to be scheduled corresponding to the second expert module copy, obtain a new fifth sequence, and return to the step of obtaining the fifth acceleration device corresponding to the current minimum second actual load in each fourth acceleration device to obtain a new fifth acceleration device. This process continues until the new first target quantity is 0, thus completing the expert module copy setting of the expert module to be scheduled.

[0066] Specifically, the target acceleration device corresponding to the position of the expert module copy to be released is designated as the fourth acceleration device in advance. The fifth acceleration device corresponding to the current minimum second actual load is obtained from the fourth acceleration device. The expert module copy to be released is released at the position of the expert module copy to be released on the fifth acceleration device. The corresponding second expert module copy is set for the expert module to be scheduled corresponding to the current maximum second sub-actual load in the fifth sequence and inserted into the released expert module copy position to move the second sub-actual load to the second expert module copy. The first target quantity is decremented by 1 to obtain the new first target quantity.

[0067] In the next scheduling, the second sub-actual load of the expert module to be scheduled corresponding to the second expert module copy is deleted, and a new fifth sequence is obtained until the first target quantity is used up.

[0068] In addition, when inserting a new copy of an expert module, the expert module copies need to be located in the same storage area during the retrieval process to reduce fragmented memory and facilitate memory management.

[0069] In this embodiment, after releasing the expert module copy position in the acceleration device corresponding to the current minimum second actual load during the current scheduling, the corresponding expert module copy position is set for the expert module to be scheduled, so as to transfer the load of the expert module without an allocated expert module copy. By releasing and adding expert module copy positions, it is ensured that a complete set of expert module types exists at any time, saving scheduling time, improving scheduling efficiency, and ensuring load balancing.

[0070] In some embodiments, the process of determining the first target quantity includes: When the number of released copies is 0, the first target number is the number of reserved additional copies under the first reserved additional copy position that is pre-set; When the number of released replicas is not zero, the process of determining the first target number specifically includes: Based on the actual number of replicas of each target expert module and the corresponding first actual load, global pre-scheduling is performed on the corresponding acceleration device to determine the planned number of replicas of each target expert module and the planned load on each target acceleration device after the planned scheduling. The number of copies is used as the first target number based on the change between the actual number of copies and the planned number of copies for each target expert module. Alternatively, the first target quantity can be determined based on the relationship between the planned load and the average second actual load, and the change in the number of actual and planned copies of each target expert module.

[0071] Specifically, the number of released replicas refers to the additional expert module replicas released, not the initial expert module replicas. This means that, at all times, the corresponding target expert module on all acceleration devices retains at least one copy of the initial expert module. When the number of released replicas is 0, an additional first reserved extra replica count is added to the existing replica count, serving as the first target count. When the number of released replicas is not 0, it can be a full release or a partial release. For partial release, there are two schemes: The first scheme directly uses the change in replica count between the planned and actual replica counts obtained from global pre-scheduling of the target expert module as the first target count to be released. The second scheme determines the first target count based on the relationship between the planned load and the average second actual load of each target acceleration device, and the change in replica count between the actual and planned replica counts.

[0072] The first target number for releasing replicas provided in this embodiment is determined when the number of replicas is 0 and when it is not 0. When it is not 0, the variable number of replicas is determined based on the planned number of replicas and the actual number of replicas after global pre-scheduling processing. This avoids the problem of poor balance caused by arbitrary release, improves the accuracy of the scheduling process, and reduces the number of scheduling operations.

[0073] In some embodiments, the number of replicas based on the change between the actual number of replicas and the planned number of replicas for each target expert module is used as the first target number, including: Compare the actual number of copies and the planned number of copies for each target expert module; If the planned number of replicas is less than the actual number of replicas, then the expert module corresponding to the planned number of replicas being less than the actual number of replicas will be used as the third target expert module. Subtract the planned number of replicas from the actual number of replicas to obtain the corresponding number of sub-released replicas; The number of sub-released copies corresponding to each of the third target expert modules is counted and used as the number of the first target.

[0074] Specifically, both the actual and planned replica counts here include the initial number of expert module replicas. The number of expert replicas for each expert is determined based on the load, number of acceleration devices, and additional replicas corresponding to each expert module currently being invoked. This number is compared to the number of expert replicas for each expert before the update. If the updated number of expert replicas for that expert is less than the original number, then the current expert replicas need to be released. In other words, if the planned replica count is less than the actual replica count, the expert module with the less planned replica count is designated as the third target expert module. Within the third target expert module, the planned replica count is subtracted from the actual replica count to obtain the corresponding sub-release replica count. Finally, the total number of sub-release replicas is used as the first target count.

[0075] For example, the previous data request had two copies of expert modules 0 and 2, and one copy of expert modules 1 and 3. The current data request has two copies of expert modules 0 and 1. When expert module 1 changes, one copy of expert module 1 is released.

[0076] This embodiment provides a method for determining the first target number based on the change in the number of copies between the actual number of copies and the planned number of copies for each target expert module. This simplifies the process of determining the number of copies to be released while improving inference efficiency.

[0077] In some embodiments, determining the first target number based on the relationship between planned load and average second actual load, and the change in the number of actual replicas and planned replicas of each target expert module, includes: The planned load on each target acceleration device is sorted in descending order to obtain the sixth sequence; In the sixth sequence, the first target acceleration device is determined based on the planned load and the average second actual load. Based on the number of first plan copies corresponding to each target expert module on the first target acceleration device; Count the number of second actual replicas corresponding to each target expert module on the first target acceleration device before load scheduling. When the number of planned copies is less than or equal to the number of actual copies, the expert copy position of the target expert module corresponding to the number of planned copies being less than or equal to the number of actual copies is retained; When the number of planned copies is greater than the number of actual copies, the expert copy positions of the target expert modules corresponding to the number of planned copies greater than the number of actual copies are released. The number of planned copies is subtracted from the number of actual copies to obtain the corresponding number of first sub-released copies. The number of first sub-released copies corresponding to the target expert modules corresponding to the number of planned copies greater than the number of actual copies is counted. The remaining target acceleration devices in the sixth sequence, excluding the first target acceleration device, are designated as the second target acceleration devices. The number of replica expert modules within the planned replica count of each of the second target acceleration devices is taken as the number of second sub-release replicas; The number of copies released by the first child and the number of copies released by the second child are used as the first target quantity.

[0078] Specifically, the steps are as follows: 1. Obtain the number of replicas ny and the load rate py for each expert module, and then calculate the average load rate avg(P) on each accelerator card (the average value is not affected by the actual expert location).

[0079] 2. Before releasing F expert modules, calculate the load rate P of each accelerator card under the original expert module arrangement, and arrange each accelerator card in ascending order of |P-avg(P)|.

[0080] 3. Select all accelerator cards where |P-avg(P)| / avg(P) < 0.15 (threshold can be adjusted), and sequentially accumulate the expert modules on each accelerator card to obtain the number of expert modules nx. If nx is less than or equal to ny for a certain expert module, then all expert modules on all accelerator cards are retained. If nx is greater than ny, then only ny expert modules are retained, and the corresponding expert modules on the accelerator cards with the next highest order need to be released. Count the number of retained expert modules nk, and then calculate the number of expert modules that can be released (at this time, the number of expert modules released is min(nx-1, ny-nk)).

[0081] 4. Based on the partial release of expert modules in step 3, release the remaining experts for all accelerator cards with |P-avg(P)| / avg(P)≥0.15 (threshold adjustable). This is based on the change in load rate on each accelerator device before and after the update. Load rate Px is the load rate on each accelerator device before the update. Load rate Py represents the load rate on each accelerator device after the update. nx is the number of expert module copies for each expert module type on all accelerator devices before the update, and ny is the number of expert module copies for each expert module type on all accelerator devices after the update. If nx is less than or equal to ny, it means that the additional expert module copies for that expert module type need to be retained. Otherwise, retain nx additional expert module copies for that expert module type after the update. Release nx-ny additional expert module copies. Which one to release depends on the load rate of the accelerator cards that is far from the average value before the update, i.e., the accelerator cards with the later load rates containing additional expert module copies for that expert module type.

[0082] For example, before the update, the number of second actual replicas for each target expert module was as follows: Expert module 0 had 3 replicas, and expert modules 1-5 each had 1 replica. After the update, the number of first planned replicas for each target expert module was as follows: Expert module 0 had 2 replicas, and expert module 5 had 2 replicas. Before the update, these were allocated to four acceleration devices: Acceleration device E included expert modules 0 and 1, Acceleration device F included expert modules 0 and 2, Acceleration device G included expert modules 0 and 3, and Acceleration device H included expert modules 4 and 5. The changes in the number of replicas for each expert module type before and after the update were statistically analyzed. Referring to the example, when the number of first planned replicas was less than or equal to the number of second actual replicas, the expert replica positions of the target expert modules whose number of first planned replicas was less than or equal to the number of second actual replicas were retained. When the number of planned replicas exceeds the number of actual replicas, the expert replica positions of the target expert modules corresponding to the larger number of planned replicas are released. The number of planned replicas is then subtracted from the number of actual replicas to obtain the corresponding number of first sub-released replicas. The number of first sub-released replicas corresponding to the target expert modules corresponding to the larger number of planned replicas is then counted. The number of replica expert modules at the expert positions within the planned replica count for each of the second target acceleration devices is then counted as the number of second sub-released replicas. The number of first sub-released replicas and the number of second sub-released replicas are then used as the first target quantity.

[0083] This embodiment specifies which expert module copies to release based on the number of released copies. This is achieved by statistically analyzing the expert module copies for each expert module type and saving the load values ​​on each accelerator card that deviate from the average value before the update, sorted by the load corresponding to the expert module. This ensures the accuracy of the released copy count while improving inference efficiency.

[0084] In some embodiments, considering that during actual execution, the acceleration device can only identify one expert module with only one corresponding position on the acceleration device, but during the invocation process, an expert module, after using expert copies, corresponds to multiple positions on the acceleration device, this results in a new expert label corresponding to one original expert label, and one original expert label corresponding to one or more new expert labels. The acceleration device excels at processing ordered labels; for out-of-order labels, recognition errors and confusion may occur, thereby reducing inference efficiency. After the target acceleration device reaches the equilibrium condition, the following is also included: An ordered mapping relationship is established between each target acceleration device, each target expert module, and the number of their respective expert module copies under the condition of reaching equilibrium. The scheduling and processing of each target expert module on the corresponding target acceleration device are performed according to the ordered mapping relationship.

[0085] Specifically, after the equilibrium condition is reached, an ordered mapping relationship is established between each target acceleration device, each target expert module, and the number of corresponding expert module copies under the equilibrium condition, so that the target acceleration device truly identifies the logical tag on one acceleration device corresponding to one expert module.

[0086] The scheduling process is performed based on the ordered mapping relationship. The specific execution process is the same as the conventional scheme and is not limited here. It can be set according to the actual situation.

[0087] The establishment of the ordered mapping relationship provided in this embodiment ensures that the target acceleration device can identify the uniqueness of the corresponding expert module on the corresponding target acceleration device in an orderly manner according to the ordered mapping relationship during the execution scheduling process, thereby avoiding identification errors and improving execution efficiency.

[0088] In some embodiments, an ordered mapping relationship is established based on the number of target acceleration devices, target expert modules, and their corresponding expert module replicas under equilibrium conditions, including: When the actual number of replicas of each target expert module is 1 when the balance condition is not met, the first physical mapping relationship of each target expert module deployed on each target acceleration device before the balance condition is not met is obtained; wherein, the first physical mapping relationship is the mapping relationship of one target expert module to one target acceleration device; Obtain the first actual number of replicas for each target expert module that has reached equilibrium. The first quantity is obtained by dividing the first actual load and the first actual replica count. The first logical marker of each target expert module on the corresponding target acceleration device is determined based on the first quantity; The first logical mapping relationship is determined based on the first physical mapping relationship and the first logical tag, so as to serve as an ordered mapping relationship; wherein, the first logical mapping relationship is a mapping relationship between one target expert module and multiple target acceleration devices.

[0089] Specifically, when there are no other additional copies under the condition of not satisfying the balance, the first physical mapping relationship established on each target acceleration device is obtained, which is the mapping relationship of one target expert module to one target acceleration device.

[0090] After achieving load balancing, the first actual replica count for each target expert module is obtained when other replicas exist. The first actual load is divided by the first actual replica count to obtain the first number. If the first number is two, mapping is performed using two new expert labels, i.e., two first logical tags. The first logical mapping relationship is determined based on the first physical mapping relationship and the first logical tags. In this first logical mapping relationship, one target expert module corresponds to multiple target devices. Here, the mapping assigns different first logical tags to each target expert module.

[0091] This embodiment provides a first quantity obtained by dividing the first actual load and the first actual replica quantity to determine the corresponding first logical tag. The first logical mapping relationship is determined by referring to the settings corresponding to the target expert module in the first physical mapping relationship and the first logical tag, which ensures the accuracy of target acceleration device identification and improves execution efficiency.

[0092] In other embodiments, an ordered mapping relationship is established based on the number of target acceleration devices, target expert modules, and their corresponding expert module replicas under the equilibrium condition, including: When the actual number of replicas of each target expert module is 1 when the balance condition is not met, obtain the expert module mapping relationship of each target expert module corresponding to the first actual number of replicas on each target acceleration device when the balance condition is met. Based on the first actual number of copies of each target expert module according to the expert module mapping relationship and the deployment location of each target acceleration device, the second logical tag of each target expert module is re-marked. A second logical mapping relationship is established based on the second logical tag and the number of target acceleration devices to serve as an ordered mapping relationship; wherein, the second logical mapping relationship is a mapping relationship in which one target expert module corresponds to multiple target acceleration devices.

[0093] Specifically, the mapping relationship of each target expert module under the balanced condition, corresponding to the first actual number of expert modules on each target expert device, is obtained. Based on this expert module mapping relationship and the deployment location on each target acceleration device, the second logical tag of each target expert module is re-marked. Then, a second logical mapping relationship is established based on the second logical tag and the number of target acceleration devices. There are multiple permutations and combinations in establishing this second logical mapping relationship; the corresponding mapping relationship can be set according to the actual situation to ensure the uniqueness of the logical tags.

[0094] This embodiment provides a second logical mapping relationship by re-marking each target expert module with a second logical tag based on the first actual number of copies of each target expert module and the deployment location of each target acceleration device according to the expert module mapping relationship. Only a one-to-one mapping relationship exists during the inference process, and the mapping execution logic takes a short time.

[0095] In some embodiments, establishing a second logical mapping relationship based on a second logical tag and the number of target acceleration devices includes: A first preset logical relationship is established based on the second logical tag corresponding to each target expert module and the number of target acceleration devices to establish the second logical mapping relationship; Multiple second-preset logical relationships are obtained by arranging and combining each of the first-preset logical relationships; The final second preset logical relationship is obtained by filtering the deployment markers of the target expert modules deployed on each target acceleration device after reaching the equilibrium condition from multiple second preset logical relationships, and is used as the second logical mapping relationship.

[0096] Specifically, considering the numerous combinations within the second logical mapping relationship, it is necessary to arrange and combine each of the first preset logical relationships to obtain multiple second preset logical relationships. Then, based on the deployment markers of the target expert modules deployed on each target acceleration device after achieving the equilibrium condition, a selection process is performed. It should be noted that this selection process ensures that expert modules originally belonging to the same acceleration device remain on the same acceleration device, which, in turn, reduces communication overhead across acceleration devices in the second preset logical relationship.

[0097] For example, suppose there is a first acceleration device and a second acceleration device, with four expert modules labeled 0, 1, 2, and 3, and two additional copies of the expert modules. Through the scheduling process described above, under balanced conditions, the first acceleration device will have expert modules 0, 1, and 2, and the second acceleration device will have expert modules 1, 2, and 3. The second logical labels would then be a1, a2, and a3, and a4, a5, and a6, respectively. At this point, a one-to-one expert mapping can be constructed: 0-a1, 1-a2, a4, 2-a3, a5, 3-a6. Based on permutations and combinations, a maximum of 1*2*2*1=4 different mapping tables can be constructed. However, in practice, so many permutations and combinations are unnecessary. Only two mapping tables are needed, meaning that each logical label is mapped at least once in each mapping table combination. Figure 5 This application provides a schematic diagram illustrating the establishment of a second logical mapping relationship, as shown in the embodiments. Figure 5 As shown, assuming that the number of calls (load) of expert modules 0-4 in a certain inference is 10, 20, 20, and 10 respectively, then using the combination of (1) and (4) or the combination of (2) and (3), half of the calls use one of the mapping tables and the other calls use another mapping table. Theoretically, after mapping, the number of calls of a1-a6 are 10, 10, 10, 10, 10, and 10 respectively, which can achieve the effect of balanced mapping. The first acceleration device and the second acceleration device each call the expert 30 times.

[0098] In this embodiment, under various permutations and combinations of the second logical mapping relationship, the deployment markers of the target expert modules deployed on each target acceleration device after reaching the equilibrium condition are still used for filtering, reducing communication across acceleration devices, so as to ensure that the corresponding logical markers are matched according to the latest balanced expert physical markers, thereby improving the scheduling and execution efficiency in the inference process.

[0099] In some embodiments, the process of establishing an ordered mapping relationship during the release process includes: Pre-obtain the second logical mapping relationship corresponding to the release process; When removing the target expert module copy position corresponding to the first target quantity, the second logical tag corresponding to the target expert module copy position is deleted from the second logical mapping relationship to obtain the third logical mapping relationship; When inserting the expert module position corresponding to the actual load of the expert module to be scheduled in the fifth sequence into the position of the released expert module copy, the fourth logical mapping relationship is updated in the third logical mapping relationship. Delete the third logical mapping relationship and retain the fourth logical mapping relationship as an ordered mapping relationship.

[0100] Specifically, during the process of releasing expert module copies, when adding new expert copies after releasing the original ones, there exists a mapping relationship between the released expert module copies and the execution of the acceleration device. Considering that the acceleration device strictly follows the mapping logic, a temporary mapping relationship is added during this process. A second logical mapping relationship is obtained. When removing the target expert module copy position corresponding to the first target quantity, the second logical marker corresponding to the target expert module copy position is deleted from the second logical mapping relationship. This creates a temporary mapping relationship after the release of the expert module copies, namely the third logical mapping relationship.

[0101] The expert module position corresponding to the actual load of the expert module to be scheduled in the fifth sequence is inserted into the position of the released expert module copy, so as to update the fourth logical mapping relationship. The update process here is the same as the embodiment of establishing the logical mapping relationship in the above embodiment, and will not be described in detail here.

[0102] This embodiment establishes a temporary third logical mapping relationship during the release of expert module copies to ensure the orderliness and accuracy of the scheduling process of the acceleration device and improve inference efficiency.

[0103] In some embodiments, the method further includes: In a hybrid expert system, when the scheduling balance parameters of the expert module meet the load scheduling conditions, an update mechanism is triggered, and the system returns to the step of receiving the current data request to obtain the new current data request and perform data processing. The scheduling balancing parameters include at least one or more parameters such as the actual number of terms in the current data request, the inference runtime, and the historical balancing results corresponding to multiple consecutive scheduling time periods before the actual scheduling time; the determination method for the expert module's scheduling balancing parameters to meet the load scheduling conditions includes at least one of the following methods: When the actual number of tokens exceeds the number of tokens in the previous current data request by multiples, the scheduling balance parameters are determined to meet the load scheduling conditions. When the inference runtime reaches the runtime threshold, it is determined that the scheduling balance parameters meet the load scheduling conditions. Based on the actual scheduling time, obtain the historical load balancing results corresponding to multiple consecutive scheduling periods before the actual scheduling time; if the load balancing conditions are not met in multiple historical load balancing results, then it is determined that the scheduling balance parameters meet the load scheduling conditions.

[0104] Specifically, the timing of the update after the current scheduling process is determined by whether the scheduling balance parameters of the expert module meet the load scheduling conditions. If they do, it means that the original load scheduling mechanism under the data request does not meet the load balance of the current data request. Therefore, it is necessary to return to step S11 for data processing.

[0105] The scheduling balance parameters may include one or more parameters such as the actual number of tokens in the current data request, the inference runtime, and the historical balance results over multiple consecutive scheduling periods prior to the actual scheduling.

[0106] If the actual number of tokens suddenly increases, exceeding multiples of the number of tokens in the previous data request, then the scheduling balance parameters are deemed to meet the load scheduling conditions. When the inference runtime reaches the runtime threshold and its scheduled update time arrives, the load scheduling conditions are deemed met. If, based on the actual scheduling time, multiple historical load balancing results do not meet the load balancing conditions, an update is required.

[0107] Based on the above embodiments, there are three ways to determine the expert module to be scheduled, involving global scheduling and local scheduling. When multiple historical balancing results are all local scheduling and the corresponding load does not meet the balancing conditions, global scheduling can be used for the scheduling of the current data request.

[0108] The expert module provided in this embodiment satisfies the load scheduling conditions by determining the scheduling balance parameters, which improves the diversity and flexibility of the trigger update method, and also ensures the stability and adaptability of the quantitative scheduling process.

[0109] In some embodiments, the actual number of expert module replicas corresponding to the expert module replicas set by the expert module to be scheduled, and the number of expert module replica locations, are integer multiples of the target acceleration devices.

[0110] Specifically, the actual number of expert module replica locations is an integer multiple of the target acceleration devices to ensure resource balance across the acceleration devices. Each acceleration device needs to handle a portion of the expert module's computational tasks. When the total number of expert module replicas is an integer multiple of the number of acceleration devices, a regular mapping from expert module replicas to acceleration devices can be achieved.

[0111] The expert scheduling module monitors the load of each expert module in real time, and the integer multiple of the number of replicas provides flexibility for dynamic scheduling. When a certain type of expert module becomes a "hot expert" with high load due to input data characteristics, the integer multiple of replicas can be evenly distributed across different acceleration devices. Furthermore, in strategies such as tiered load balancing, the integer multiple configuration allows the expert scheduling module to quickly adjust the distribution of expert module replicas within and between nodes on acceleration devices, avoiding chaotic situations where some acceleration devices have no corresponding replicas or where replicas are clustered together. Setting the number of expert module replicas to an integer multiple of the number of acceleration devices allows the same group of experts or related replicas to be concentrated within the corresponding device cluster, reducing data transmission across devices and nodes.

[0112] In this embodiment, the actual number of expert module replicas corresponding to the set expert module replicas and the number of expert module replica locations are integer multiples of the target acceleration device. This avoids resource waste caused by the mismatch between the set number of expert module replicas and the acceleration device, and makes the computing pressure of the acceleration device tend to be balanced.

[0113] In some embodiments, the number of second logical mapping relationships is an integer multiple of the number of the top M target expert modules within the selection scores of the current data request in the gating network input of the corresponding hybrid expert system, sorted from high to low.

[0114] Specifically, the hybrid expert architecture comprises a gating network and multiple expert sub-networks. During inference, the gating network calculates the activation scores of each expert based on the current input and selects the top M target expert modules (i.e., Top-M experts) with the highest scores. Only the forward computation of the selected target expert modules is activated and executed, while the remaining target expert modules remain inactive. Here, M is a pre-defined positive integer, typically satisfying 1 ≤ M ≤ L, and L is the total number of expert modules. This Top-M sparse activation mechanism significantly reduces computational overhead while preserving the model's expressive power.

[0115] Selecting the top M target expert modules with the highest scores can be achieved as follows: sort the probability distribution of the gated output, select the indexes of the top M expert modules with the highest scores, and normalize the corresponding routing weights for weighted fusion of expert outputs.

[0116] The second logical mapping relationship can be represented by a mapping table.

[0117] The second mapping logic relationship setting provided in this embodiment results in a shorter mapping execution time, which improves both execution efficiency and inference efficiency.

[0118] Furthermore, this application also provides a question-answering method based on a hybrid expert system, including: Obtain the question-and-answer dataset corresponding to the reasoning question; The question-and-answer dataset is input into a hybrid expert system to output response information; The hybrid expert system is obtained by reasoning and optimization of the steps of the data processing method based on the acceleration device described above.

[0119] Specifically, the question-answering dataset is designed for specific question-answering tasks and uses accelerated devices to achieve parallel computing. While ensuring the accuracy of responses, it solves the pain points of slow inference speed and inability to handle high concurrency of single models, and adapts to the differentiated needs of different types of question-answering.

[0120] For an introduction to the question-answering method based on a hybrid expert system provided in this application, please refer to the above method embodiments. This application will not repeat the details here, but it has the same beneficial effects as the above data processing method based on acceleration devices.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0122] Embodiments of this application also provide a data processing apparatus based on an acceleration device. Figure 6 A structural diagram of a data processing apparatus based on an acceleration device provided in this application embodiment is shown below. Figure 6 As shown, the device includes: The receiving module 11 is used to receive the current data request and determine the target expert module that corresponds to the current data request and is deployed on the target acceleration device; The acquisition module 12 is used to acquire the first actual load of each target expert module and the second actual load of each target acceleration device. The determination module 13 is used to determine whether the target acceleration device meets the balancing condition based on each second actual load. The setting module 14 is used to determine the expert module to be scheduled based on the first actual load of each target expert module if the balancing condition is not met, and to set a corresponding expert module copy for the expert module to be scheduled so that the target acceleration device that does not meet the balancing condition can reach the balancing condition.

[0123] For a description of the features in the corresponding embodiment of the device, please refer to the relevant description of the corresponding embodiment of the data processing method based on the acceleration device described above, which will not be repeated here.

[0124] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the data processing method based on an acceleration device.

[0125] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the data processing method based on an acceleration device when it is run.

[0126] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0127] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the data processing method based on an acceleration device.

[0128] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the data processing method based on an acceleration device.

[0129] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0130] The foregoing has provided a detailed description of a data processing method and question-answering method based on an acceleration device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data processing method based on an acceleration device, characterized in that, include: Receive the current data request and determine the target expert module that corresponds to the current data request and is deployed on the target acceleration device; Obtain the first actual load of each of the target expert modules and the second actual load of each of the target acceleration devices; Determine whether the target acceleration device meets the balance condition based on each of the second actual loads; If the equilibrium condition is not met, then the expert module to be scheduled is determined based on the first actual load of each of the target expert modules, and a corresponding expert module copy is set for the expert module to be scheduled, so that the target acceleration device that does not meet the equilibrium condition can reach the equilibrium condition.

2. The data processing method based on an acceleration device according to claim 1, characterized in that, Determining whether the target acceleration equipment meets the balancing conditions based on the actual load of each facility includes: In the current scheduling process, the average second actual load is determined based on the current second actual load corresponding to each of the target acceleration devices. Determine the current maximum second actual load within each current second actual load; The current load balance is determined based on the current maximum second actual load and the average second actual load. If the current balance exceeds the preset balance, it is determined that each of the target acceleration devices does not meet the balance condition.

3. The data processing method based on an acceleration device according to claim 2, characterized in that, The expert modules to be scheduled are determined based on the first actual load of each of the target expert modules, including: Get the preset number of scheduling attempts; In the current scheduling count, the second actual load of each target acceleration device is sorted in descending order to obtain the first sequence; The first average load is determined based on the current maximum second actual load and the current minimum second actual load in the first sequence; If the difference between the first average load and the average second actual load is less than or equal to the first threshold, then the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load are determined to meet the scheduling conditions. The target expert module corresponding to the current maximum first actual load in the target acceleration device corresponding to the current maximum second actual load is taken as the current first expert module to be scheduled. The reference load is determined based on the first average load, the current maximum first actual load of the first expert module to be scheduled, and the current maximum second actual load. Within the first actual load of each target expert module in the target acceleration device corresponding to the current minimum second actual load, select the first actual load with the smallest difference from the reference load as the target first actual load; take the target expert module corresponding to the target first actual load as the current second expert module to be scheduled; record the current scheduling count; After determining the target acceleration devices after scheduling, return to the step of sorting the second actual load of each target acceleration device in descending order to obtain the first sequence, obtain the new first sequence to determine the expert module to be scheduled, and so on, until the current scheduling number reaches the preset scheduling number, and complete the process of determining the expert module to be scheduled.

4. The data processing method based on an acceleration device according to claim 3, characterized in that, Setting up a corresponding expert module copy for the expert module to be scheduled includes: In the current scheduling count, the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load that meet the scheduling conditions are respectively designated as the current first acceleration device and the current second acceleration device. Establish temporary expert module copy locations on the current first acceleration device and the current second acceleration device; At the temporary expert module copy location on the current second acceleration device, a corresponding first expert module copy is set for the current first expert module to be scheduled, so as to fully process the first actual load corresponding to the current first expert module to be scheduled to the first expert module copy, and delete the expert module location corresponding to the current first expert module to be scheduled on the current first acceleration device. At the temporary expert module copy location on the current first acceleration device, a corresponding second expert module copy is set for the current second expert module to be scheduled, so as to process the first actual load corresponding to the current second expert module to be scheduled in full to the second expert module copy, and delete the expert module location corresponding to the current second expert module to be scheduled on the current second acceleration device. Increment the current scheduling count by 1 to obtain a new current scheduling count, and return to the step of setting the target acceleration devices corresponding to the current maximum second actual load and the current minimum second actual load that meet the scheduling conditions as the current first acceleration device and the current second acceleration device, respectively. Continue in this manner until the new current scheduling count reaches the preset scheduling count, and complete the process of setting up the expert module replica of the expert module to be scheduled.

5. The data processing method based on an acceleration device according to claim 2, characterized in that, The expert modules to be scheduled are determined based on the first actual load of each of the target expert modules, including: Get the number of pre-reserved expert module copy locations; The second sequence is obtained by sorting the first actual load of each target expert module in descending order. In the second sequence, the first target expert module corresponding to the current maximum first actual load is taken as the expert module to be scheduled, so as to perform balancing processing on the expert module to be scheduled to obtain the first sub-actual load, and the number of expert module replica positions is reduced by 1 to obtain the new number of expert module replicas. The first sub-actual load and the first actual load corresponding to other target expert modules (excluding the first target expert module) are sorted in descending order to obtain a new second sequence. Then, the process proceeds to the step of selecting the first target expert module corresponding to the current largest first actual load in the second sequence as the expert module to be scheduled, thus obtaining a new expert module to be scheduled. This process is repeated until the number of new expert module replicas is 0, so as to obtain the corresponding expert module to be scheduled.

6. The data processing method based on an acceleration device according to claim 5, characterized in that, When no expert module replica exists for the other non-scheduled expert modules of the expert module to be scheduled, a corresponding expert module replica is set for the expert module to be scheduled, including: The third sequence is obtained by sorting the first sub-actual load of each expert module to be scheduled in descending order. Obtain the third acceleration device corresponding to the current minimum second actual load; At the expert module replica position on the third acceleration device, set the corresponding first expert module replica for the expert module to be scheduled in the third sequence that corresponds to the current maximum first sub-actual load, so as to move the first sub-actual load to the first expert module replica, and decrement the number of expert module replica positions by 1 to obtain a new number of expert module replicas. In the third sequence, delete the first sub-actual load of the expert module to be scheduled corresponding to the first expert module replica, obtain a new third sequence, and return to the step of obtaining the third acceleration device corresponding to the current minimum second actual load, so as to obtain a new third acceleration device. This process is repeated until the number of new expert module replica positions is 0, thus completing the expert module replica setting of the expert module to be scheduled.

7. The data processing method based on an acceleration device according to claim 2, characterized in that, The expert modules to be scheduled are determined based on the first actual load of each of the target expert modules, including: Obtain a first target number of expert module copy locations to be released; wherein, the first target number is less than or equal to the number of existing expert module copies for each target expert module; Obtain the second target expert module corresponding to the number of expert module copies that do not exist in each target expert module; The fourth sequence is obtained by sorting the first actual load of each second objective expert module in descending order. In the fourth sequence, the second target expert module corresponding to the current maximum first actual load is taken as the expert module to be scheduled, so as to perform balancing processing on the expert module to be scheduled to obtain the second sub-actual load, and the first target quantity is reduced by 1 to obtain the new first target quantity. The actual load of each second sub-sub and the first actual load corresponding to other second target expert modules in the second target expert module (excluding the scheduled expert module) are sorted in descending order to obtain a new fourth sequence. Then, the process returns to the step of selecting the second target expert module corresponding to the current largest first actual load in the fourth sequence as the scheduled expert module, and so on, until the number of new first targets is 0, so as to obtain the corresponding scheduled expert modules.

8. The data processing method based on an acceleration device according to claim 7, characterized in that, When there are expert module copies for the other non-scheduled expert modules of the expert module to be scheduled, a corresponding expert module copy is set for the expert module to be scheduled, including: Pre-select the target acceleration devices corresponding to the locations of the expert module copies to be released as the fourth acceleration devices; Find the fifth acceleration device corresponding to the current minimum second actual load among all the fourth acceleration devices; The fifth sequence is obtained by sorting the second sub-actual load of each expert module to be scheduled in descending order. Release the expert module copy on the fifth acceleration device, set the corresponding second expert module copy for the expert module to be scheduled corresponding to the current maximum second sub-actual load in the fifth sequence, and insert it into the released expert module copy position to move the second sub-actual load to the second expert module copy, and decrement the first target quantity by 1 to obtain a new first target quantity; In the fifth sequence, delete the second sub-actual load of the expert module to be scheduled corresponding to the second expert module copy, obtain a new fifth sequence, and return to the step of obtaining the fifth acceleration device corresponding to the current minimum second actual load in each fourth acceleration device to obtain a new fifth acceleration device. This process continues until the new first target quantity is 0, thus completing the expert module copy setting of the expert module to be scheduled.

9. The data processing method based on an acceleration device according to any one of claims 1 to 8, characterized in that, After the target acceleration device reaches the equilibrium condition, the following is also included: An ordered mapping relationship is established between each target acceleration device, each target expert module, and the number of their respective expert module copies under the condition of reaching equilibrium. According to the ordered mapping relationship, the scheduling processing of each target expert module on the corresponding target acceleration device is performed.

10. The data processing method based on an acceleration device according to claim 9, characterized in that, An ordered mapping relationship is established based on the number of target acceleration devices, target expert modules, and their corresponding expert module replicas under equilibrium conditions, including: When the actual number of replicas of each of the target expert modules is 1 when the balance condition is not met, the first physical mapping relationship established by each of the target expert modules on each of the target acceleration devices before the balance condition is not met is obtained; wherein, the first physical mapping relationship is a mapping relationship of one target expert module to one target acceleration device; Obtain the first actual number of replicas corresponding to each of the target expert modules under the equilibrium condition; The first quantity is obtained by dividing the first actual load and the first actual number of replicas. Based on the first quantity, determine the first logical marker of each target expert module on the corresponding target acceleration device; A first logical mapping relationship is determined based on the first physical mapping relationship and the first logical tag, to serve as the ordered mapping relationship; wherein, the first logical mapping relationship is a mapping relationship between one target expert module and multiple target acceleration devices.

11. The data processing method based on an acceleration device according to claim 9, characterized in that, An ordered mapping relationship is established based on the number of target acceleration devices, target expert modules, and their corresponding expert module replicas under equilibrium conditions, including: When the actual number of replicas of each of the target expert modules that do not meet the equilibrium condition is 1, obtain the expert module mapping relationship of each of the target expert modules that meet the equilibrium condition on each of the target acceleration devices corresponding to the first actual number of replicas. Based on the expert module mapping relationship, the first actual number of copies of each target expert module and the deployment location of each target acceleration device, the second logical tag of each target expert module is re-marked; A second logical mapping relationship is established based on the second logical tag and the number of target acceleration devices to serve as the ordered mapping relationship; wherein, the second logical mapping relationship is a mapping relationship in which one target expert module corresponds to multiple target acceleration devices.

12. The data processing method based on an acceleration device according to claim 8, characterized in that, The process of determining the first target quantity includes: When the number of released copies is 0, the first target number is determined by the number of reserved additional copies under the first reserved additional copy position that is pre-set; When the number of released replicas is not zero, the process of determining the first target number specifically includes: Based on the actual number of replicas of each target expert module and the corresponding first actual load, global pre-scheduling processing is performed on the corresponding acceleration device to determine the planned number of replicas of each target expert module and the planned load on each target acceleration device after the planned scheduling. The first target number is the number of replicas that changes between the actual number of replicas and the planned number of replicas for each of the target expert modules. Alternatively, the first target quantity can be determined based on the relationship between the planned load and the average second actual load, and the change in the number of actual copies of each target expert module and the planned copy quantity.

13. The data processing method based on an acceleration device according to claim 12, characterized in that, The first target number is the number of replicas that varies between the actual number of replicas and the planned number of replicas for each of the target expert modules, including: Compare the actual number of copies of each of the target expert modules with the planned number of copies; If the planned number of replicas is less than the actual number of replicas, then the expert module corresponding to the planned number of replicas being less than the actual number of replicas will be used as the third target expert module. Subtracting the planned number of replicas from the actual number of replicas yields the corresponding number of sub-released replicas; The number of sub-released copies corresponding to each of the third target expert modules is counted and used as the first target quantity.

14. The data processing method based on an acceleration device according to claim 12, characterized in that, The first target quantity is determined based on the relationship between the planned load and the average second actual load, and the change in the number of actual replicas of each target expert module compared to the planned replica quantity, including: The planned load on each of the target acceleration devices is sorted in descending order to obtain the sixth sequence; In the sixth sequence, a first target acceleration device is determined based on the planned load and the average second actual load. Based on the number of first plan copies corresponding to each target expert module on the first target acceleration device; Count the number of second actual replicas of each target expert module on the first target acceleration device before load scheduling. When the number of planned copies is less than or equal to the number of actual copies, the expert copy position of the target expert module corresponding to the number of planned copies being less than or equal to the number of actual copies is retained; When the number of planned replicas is greater than the number of actual replicas, the expert replica positions of the target expert modules corresponding to the number of planned replicas being greater than the number of actual replicas are released, and the number of planned replicas is subtracted from the number of actual replicas to obtain the corresponding number of first sub-released replicas; the number of first sub-released replicas corresponding to the target expert modules corresponding to the number of planned replicas being greater than the number of actual replicas is counted. The remaining target acceleration devices in the sixth sequence, excluding the first target acceleration device, are designated as the second target acceleration devices. The number of replica expert modules at the replica expert positions within the planned replica count for each of the second target acceleration devices is taken as the number of second sub-release replicas. The number of the first sub-released copies and the number of the second sub-released copies are used as the first target number.

15. The data processing method based on an acceleration device according to claim 1, characterized in that, The method further includes: When the scheduling balance parameters of the expert module in the hybrid expert system meet the load scheduling conditions, the update mechanism is triggered, and the process returns to the step of receiving the current data request to obtain the new current data request and perform data processing. The scheduling balancing parameters include at least one or more parameters such as the actual number of tokens in the current data request, the inference runtime, and the historical balancing results corresponding to multiple consecutive scheduling time periods before the actual scheduling time; the determination method for the expert module's scheduling balancing parameters to satisfy the load scheduling conditions includes at least one of the following methods: When the actual number of tokens is a multiple of the number of tokens in the previous current data request, it is determined that the scheduling balance parameter meets the load scheduling condition. When the inference runtime reaches the runtime threshold, it is determined that the scheduling balance parameters meet the load scheduling conditions. Based on the actual scheduling time, obtain the historical load balancing results corresponding to multiple consecutive scheduling periods before the actual scheduling time; if the load balancing conditions are not met in multiple historical load balancing results, then determine that the scheduling balance parameters meet the load scheduling conditions.

16. The data processing method based on an acceleration device according to claim 11, characterized in that, The process of establishing an ordered mapping relationship during the release includes: Pre-obtain the second logical mapping relationship corresponding to the release process; When removing the target expert module copy position corresponding to the first target quantity, the second logical tag corresponding to the target expert module copy position is deleted from the second logical mapping relationship to obtain the third logical mapping relationship; When the expert module position corresponding to the actual load of the expert module to be scheduled in the fifth sequence is inserted into the position of the released expert module copy, the fourth logical mapping relationship is updated in the third logical mapping relationship. The third logical mapping relationship is deleted, and the fourth logical mapping relationship is retained as the ordered mapping relationship.

17. The data processing method based on an acceleration device according to any one of claims 1 to 8, characterized in that, The actual number of expert module replicas corresponding to the expert module replicas set by the expert module to be scheduled is an integer multiple of the target acceleration device.

18. The data processing method based on an acceleration device according to claim 11, characterized in that, Establishing a second logical mapping relationship based on the second logical tag and the number of target acceleration devices includes: A first preset logical relationship is established based on the second logical tag corresponding to each of the target expert modules and the number of target acceleration devices to establish a second logical mapping relationship; Multiple second preset logical relationships are obtained by arranging and combining each of the first preset logical relationships; The final second preset logical relationship is obtained by filtering the deployment tags of the target expert modules deployed on each target acceleration device after reaching the equilibrium condition among multiple second preset logical relationships, and is used as the second logical mapping relationship.

19. The data processing method based on an acceleration device according to claim 11, characterized in that, The number of the second logical mapping relationship is an integer multiple of the number of the top M target expert modules in the selection score sorted from high to low in the gating network input of the current data request in the corresponding hybrid expert system.

20. A question-answering method based on a hybrid expert system, characterized in that, include: Obtain the question-and-answer dataset corresponding to the reasoning question; The question-and-answer dataset is input into a hybrid expert system to output response information; The hybrid expert system is obtained by reasoning optimization through the steps of the data processing method based on acceleration device as described in any one of claims 1 to 19.

Images