A distributed scheduling method based on split federated learning

Abstract

The present invention discloses a distributed scheduling method based on split federated learning, and relates to the technical field of electronic digital data processing. The distributed scheduling method based on split federated learning includes the following steps: obtaining client status information; performing matching evaluation; performing task allocation; and performing dynamic optimization. The present invention performs matching evaluation on client status information, sample information, and task data, screens out a client group to be tested according to the obtained resource matching index, performs task allocation to obtain a task queue, and then transmits the task queue to the client group to be tested according to a transmission strategy. Finally, dynamic optimization is performed according to the task execution evaluation index, thereby achieving a more stable distributed resource scheduling effect, and solving the problem of low scheduling stability in the distributed scheduling process based on split federated learning in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a distributed scheduling method based on split federated learning. Background Art

[0002] With the rapid development of big data, artificial intelligence, and the Internet of Things, more and more application scenarios need to process massive amounts of data and extract valuable information from them. This demand has led to the widespread application of distributed computing, especially in areas such as federated learning that require the integration of distributed data resources. Federated learning distributes model training tasks to multiple devices or nodes, which not only improves computing efficiency but also effectively protects data privacy. However, as the scale of the system expands and the computing tasks become more complex, how to effectively coordinate computing tasks between multiple nodes has become a key challenge to improving the performance of distributed systems. This makes distributed scheduling methods particularly important in modern distributed computing frameworks and a core mechanism to ensure efficient operation of the system.

[0003] Existing distributed scheduling methods usually manage and allocate computing tasks through a central control node or a decentralized coordination mechanism. These methods dynamically adjust task allocation by analyzing the computing power, network status, and task requirements of each node to achieve load balancing. Some scheduling algorithms also take into account the communication overhead between nodes, and reduce the overall network latency and bandwidth consumption of the system by optimizing task division and scheduling order. In addition, some methods introduce a prediction mechanism to estimate the performance of each node through historical data analysis to further optimize scheduling decisions. These methods have improved the resource utilization and computing efficiency of distributed systems to a certain extent.

[0004] For example, the invention patent with announcement number: CN112860396B announces a GPU scheduling method and system based on distributed deep learning, including: creating tasks to the scheduling center, and the tasks are placed in the task queue in sequence according to priority, waiting to be scheduled; the scheduling center periodically retrieves tasks from the task queue, and queries the node selector for idle working nodes based on the GPU resources required for the tasks; after the node selector queries for idle working nodes among multiple working nodes, it feeds back the node information of the idle working nodes to the scheduling center; the scheduling center combines the task and node information into a comprehensive task information and puts it into the task pool; each node monitors the task pool in real time, and when the working node detects that there are executable tasks in the task pool, it retrieves the corresponding comprehensive task information for execution.

[0005] For example, the distributed scheduling method, apparatus, device and storage medium disclosed in the patent application with publication number: CN113961318A include: sending registration applications of nodes of multiple participants to a central coordinator; regularly sending heartbeat information to the central coordinator, wherein the heartbeat information includes the current system resource status and job status of the nodes of the multiple participants; starting a resource container on the node of the target participant, and jointly running the job with the nodes of other participants.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] In the existing technology, although the distributed scheduling method can improve the system efficiency to a certain extent, due to the differences in the computing power, network status and task load of each node, it may make it difficult for the scheduling strategy to maintain the global consistency and efficiency of the system in a dynamic environment. There is a problem of low scheduling stability in the distributed scheduling process based on split federated learning. Summary of the invention

[0008] The embodiment of the present application solves the problem of low scheduling stability in the distributed scheduling process based on split federated learning in the prior art by providing a distributed scheduling method based on split federated learning, and realizes more stable distributed resource scheduling.

[0009] The embodiment of the present application provides a distributed scheduling method based on split federated learning, comprising the following steps: S1, obtaining sample information and client status information of an initial client group to be tested; S2, performing matching evaluation based on the client status information, sample information and task data obtained from a preset database to obtain a resource matching index of the initial client group to be tested, screening the initial client group to be tested based on the resource matching index to obtain a client group to be tested, wherein the resource matching index is used to describe the degree of satisfaction of the computing resources, network transmission capabilities and data sets of different clients to be tested for the training task; S3, performing task allocation for the training task based on the resource matching index of the client group to be tested to obtain a task queue, performing transmission analysis on the client status information to obtain a transmission strategy for the client group to be tested, and transmitting the task queue to the client group to be tested through the transmission strategy; S4, receiving task execution data sent by the client group to be tested, performing task evaluation based on the task execution data to obtain a task execution evaluation index of the client group to be tested, performing dynamic optimization based on the task execution evaluation index and a preset task evaluation threshold until the training of the task to be trained is completed, wherein the task execution evaluation index is used to quantitatively describe the comprehensive performance of the client to be tested when executing the task.

[0010] Further, the method of obtaining sample information and client status information of the initial client group to be tested also includes filtering out the initial client group to be tested from the client pool, and the specific steps are as follows: A1, obtaining historical client status information of all clients to be tested in the client pool from a preset database, wherein the historical client status information includes historical CPU occupancy, historical memory occupancy, historical remaining power, historical packet loss rate, historical network delay and historical bandwidth; A2, filtering out a candidate client group according to the historical client status information and a preset status threshold obtained from the preset database; A3, sending request response information to the candidate client group respectively, if no response information is received within a preset time period, deleting the corresponding client to be tested from the candidate client group, otherwise sorting the candidate client group according to the response time after the response time of the client to be tested is recorded; A4, judging whether the total number of clients to be tested in the candidate client group is not less than the preset first number of clients, if not less than the preset first number of clients, extracting the preset first number of clients to be tested from the candidate clients and recording them as the initial client group to be tested, otherwise sending a prompt message of missing clients to the preset personnel.

[0011] Furthermore, the resource matching index of the initial client group to be tested is obtained by performing a matching evaluation based on the client status information, sample information and task data obtained from a preset database, and the process also includes performing a status evaluation on the client status information of the client to be tested to obtain a real-time status index; the real-time status index is used to describe the currently available computing resources and network transmission capabilities of different clients to be tested; the specific process of performing a status evaluation on the client status information of the client to be tested to obtain the real-time status index is as follows: obtaining status index evaluation weights from a preset database, the status index evaluation weights including a first status weight, a second status weight, a third status weight, a fourth status weight, a fifth status weight and a sixth status weight; obtaining the real-time status index of the client to be tested based on the client status information, the status index evaluation weights and a preset status threshold, the client status information including computing resource data and network transmission data, the computing resource data including CPU occupancy, memory occupancy and remaining power, and the network transmission data including network delay, packet loss rate and bandwidth.

[0012] Furthermore, the specific process of performing matching evaluation according to the client status information, sample information and task data obtained from the preset database to obtain the resource matching index of the initial client group to be tested is as follows: obtaining matching evaluation weights from the preset database, the matching evaluation weights including the first matching weight, the second matching weight and the third matching weight; obtaining the resource matching index of the client to be tested through the real-time status index, sample information, task data and matching evaluation weights, the sample information including the number of samples and the number of sample dimensions, the task data including the model parameter quantity, the number of model layers, the number of input samples and the number of input sample dimensions, and the specific calculation formula of the resource matching index is as follows:

[0013] ;

[0014] Where n is the number of the client to be tested in the initial client group to be tested, , N is the preset number of first clients, is the real-time status index of the nth client to be tested in the initial client group to be tested, is the resource matching index of the nth client to be tested in the initial client group to be tested, is the number of samples of the nth client to be tested in the initial client group to be tested, is the number of sample dimensions of the nth client to be tested in the initial client group to be tested, is the number of input samples of the task to be trained, is the number of dimensions of the input samples of the task to be trained, is the model parameter of the task to be trained, is the number of model layers of the task to be trained, is the first matching weight, is the second matching weight, is the third matching weight, is the complexity adjustment factor.

[0015] Furthermore, the specific steps of screening the initial client group to be tested according to the resource matching index to obtain the client group to be tested are as follows: B1, obtaining a preset matching threshold from a preset database; B2, traversing the initial client group to be tested, and determining whether the resource matching index of the client to be tested is not lower than the preset matching threshold; if it is not lower than the preset matching threshold, recording the corresponding client to be tested into the client group to be selected, otherwise no processing is performed; B3, sorting the client group to be selected in order according to the resource matching index; B4, determining whether the total number of clients to be tested in the client group to be selected is not less than a preset second number of clients; if it is not less than the preset second number of clients, extracting the preset second number of clients to be tested from the client group to be selected and recording them as the client group to be tested, otherwise sending a prompt message of missing clients to a preset person.

[0016] Further, the specific steps of the task allocation are as follows: C1, the resource matching index of the client group to be tested is counted to obtain the total resource matching index, and the resource weight of the corresponding client to be tested is obtained in combination with the resource matching index, the total resource matching index represents the sum of the resource matching index of all the clients to be tested in the client group to be tested, and the resource weight represents the ratio of the resource matching index of the client to be tested to the total resource matching index; C2, the nth round of training tasks are counted to obtain the total number of task blocks and task block numbers of the nth round, and the number of task blocks corresponding to the client to be tested is obtained in combination with the resource weight; C3, traverse the client group to be tested, and determine whether the number of task blocks corresponding to the client to be tested is greater than a preset task block threshold. If it is greater than the preset task block threshold, the task block number of the preset task block threshold is written into the task queue, otherwise the task block number of the number of task blocks is stored in the task queue, and the preset task block threshold is determined by the preset personnel according to the training efficiency of the distributed scheduling method; C4, determine whether there is a task block in the nth round of training tasks that is not stored in the task queue. If so, the task block is merged into the n+1th round of training tasks, otherwise it is not processed.

[0017] Furthermore, the specific steps of performing transmission analysis on the client status information to obtain the transmission strategy of the client group to be tested are as follows: D1, judging whether the bandwidth of the client to be tested is not greater than the bandwidth selection threshold, if it is not greater than the bandwidth selection threshold, selecting the UDP protocol as the transmission protocol, otherwise selecting the TCP protocol as the transmission protocol; D2, judging whether the network delay of the client to be tested is not greater than the delay selection threshold, if it is not greater than the delay selection threshold, selecting no compression as the compression strategy, otherwise selecting lossy compression as the compression strategy; D3, judging whether the packet loss rate of the client to be tested is not greater than the packet loss rate selection threshold, if it is not greater than the packet loss rate selection threshold, selecting a preset first number as the transmission number, otherwise selecting a preset second number as the transmission number.

[0018] Furthermore, the specific process of performing task evaluation based on task execution data to obtain a task execution evaluation index for the client group to be tested is as follows: obtaining task evaluation weights from a preset database, wherein the task evaluation weights include a first task execution weight, a second task execution weight, and a third task execution weight; obtaining expected task data from a preset database, wherein the expected task data include a model error rate reference value, a training loss value reference value, and a task completion time reference value; obtaining the task execution evaluation index of the client to be tested through the task execution data, the expected task data, and the task evaluation weights, wherein the task execution data include a model error rate, a training loss value, and a task completion time.

[0019] Furthermore, the specific process of dynamic optimization based on the task execution evaluation index and the preset task evaluation threshold is as follows: traverse the client group to be tested, determine whether the task execution evaluation index of the client to be tested is not less than the preset task evaluation threshold, if it is not less than the preset task evaluation threshold, no processing is performed, otherwise the corresponding task block is marked as an unfinished task block, and the resource matching index of the corresponding client to be tested is recalculated; after the traversal is completed, the unfinished task block is retrained, the client group to be tested is sorted in order according to the resource matching index, and task allocation is performed.

[0020] Furthermore, the dynamic optimization is performed according to the task execution evaluation index and the preset task evaluation threshold until the training of the task to be trained is completed, and then the client visualization is also included. The specific process is as follows: a real-time status index bar graph is drawn according to the change relationship between the real-time status index and the number of the client to be tested in the initial client group to be tested; a resource matching index bar graph is drawn according to the change relationship between the resource matching index and the number of the client to be tested in the initial client group to be tested; a task evaluation scatter plot is drawn according to the relationship between the task execution evaluation index and the number of the client to be tested in the client group to be tested.

[0021] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0022] 1. By evaluating the matching degree of client status information, sample information and task data, the client group to be tested is screened out according to the obtained resource matching degree index, and the task queue is obtained by task allocation. Then, the task queue is transmitted to the client group to be tested according to the transmission strategy. Finally, dynamic optimization is performed according to the task execution evaluation index, thereby improving the execution efficiency of the task to be trained, thereby achieving more stable distributed resource scheduling, and effectively solving the problem of low scheduling stability in the distributed scheduling process based on split federated learning in the prior art;

[0023] 2. The candidate client group is screened out from the client pool through historical client status information and a preset status threshold obtained from a preset database, and the initial client group to be tested is obtained based on whether a response message is received within a preset time period after sending information to the candidate client group. Finally, the initial client group to be tested is screened in combination with a preset matching degree threshold to obtain the client group to be tested, thereby improving the reliability of the selection of the client to be tested, and further improving the effectiveness of task execution in the distributed scheduling process;

[0024] 3. By receiving the task execution data sent by the client to be tested, the task execution evaluation index is obtained by combining the expected task data and task evaluation weight obtained from the preset database, and then the client to be tested is screened according to the preset task evaluation threshold to obtain the unfinished task blocks, the unfinished task blocks are retrained, and the resource matching index of the corresponding client to be tested is recalculated. Finally, tasks are allocated according to the reordered client group to be tested, thereby improving the success rate of task execution and achieving the improvement of the fault tolerance of the distributed scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a distributed scheduling method based on split federated learning provided in an embodiment of the present application;

[0026] Figure 2 A variable relationship diagram of the resource matching index provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiment of the present application solves the problem of low scheduling stability in the distributed scheduling process based on split federated learning in the prior art by providing a distributed scheduling method based on split federated learning. The method performs a matching evaluation on the acquired client status information, sample information and task data, and then screens the initial client group to be tested according to the obtained resource matching index to obtain the client group to be tested, and performs task allocation on the training task to obtain a task queue. Then, the task queue is transmitted to the client group to be tested through the transmission strategy obtained by transmission analysis. Finally, a task evaluation is performed based on the received task execution data to obtain a task execution evaluation index. Dynamic optimization is performed in combination with a preset task evaluation threshold until the training of the task to be trained is completed, thereby achieving more stable distributed resource scheduling.

[0028] The technical solution in the embodiment of the present application is to solve the problem of low scheduling stability in the distributed scheduling process based on split federated learning. The overall idea is as follows:

[0029] By evaluating the matching degree of client status information, sample information and task data to screen out the client group to be tested, and then transmitting the task queue obtained by task allocation to the client group to be tested according to the transmission strategy, and finally performing dynamic optimization according to the task execution evaluation index, a more stable distributed resource scheduling effect is achieved.

[0030] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] like Figure 1As shown, it is a flowchart of a distributed scheduling method based on split federated learning provided by an embodiment of the present application, and the method includes the following steps: S1, obtaining sample information and client status information of an initial client group to be tested, the initial client group to be tested represents a collection of clients to be tested, the sample information represents the number and feature information of local data sets used by different clients to be tested in the initial client group to be tested for the task to be trained, and the client status information represents the load, memory, power and network status of different clients to be tested in the initial client group to be tested; S2, performing a matching evaluation on the client status information, the sample information and the task data obtained from a preset database to obtain a resource matching index of the initial client group to be tested, screening the initial client group to be tested according to the resource matching index to obtain the client group to be tested, the task data represents the number of samples, feature dimensions and model parameter information of the task to be trained, and the resource matching index is used to describe different The computing resources, network transmission capacity and data set of the client under test satisfy the requirements of the training task; S3, assigning the training task to the client under test according to the resource matching index of the client group under test to obtain the task queue, and performing transmission analysis on the client status information to obtain the transmission strategy of the client group under test, and transmitting the task queue to the client group under test according to the transmission strategy, and the transmission strategy indicates the transmission method selected according to the network conditions of different clients under test; S4, receiving the task execution data sent by the client group under test, and performing task evaluation according to the task execution data to obtain the task execution evaluation index of the client group under test, and performing dynamic optimization according to the task execution evaluation index and the preset task evaluation threshold until the training of the task under test is completed, and the task execution data indicates the model update information generated by different clients under test after completing the task block, and the task execution evaluation index is used to quantitatively describe the comprehensive performance of the client under test when executing the task.

[0032] In this embodiment, the setting of the preset task evaluation threshold should be determined according to the task execution evaluation index of the historical time period. For example, by obtaining the task execution evaluation index of the historical time period from the preset database and performing statistics to obtain the mean and standard deviation, the preset task evaluation threshold is set to the mean of the task execution evaluation index plus twice the standard deviation. The twice standard deviation is determined according to the 68-95-99.7 rule (also known as the "rule of thumb"), that is, under a normal distribution, about 68% of the data falls within one standard deviation of the mean, 95% of the data falls within two standard deviations of the mean, and 99.7% of the data falls within three standard deviations of the mean. By using twice the standard deviation to determine the task execution evaluation index outside the 95% probability, the client to be tested that significantly deviates from the task execution evaluation index is screened out, thereby achieving more stable distributed resource scheduling, and effectively solving the problem of low scheduling stability in the distributed scheduling process based on split federated learning in the prior art.

[0033] Further, sample information and client status information of the initial client group to be tested are obtained, and the initial client group to be tested is previously screened out from the client pool, where the client pool represents a set of all clients to be tested that may participate in the federated learning task that are currently available for selection. The specific steps are as follows: A1, historical client status information of all clients to be tested in the client pool is obtained from a preset database, and the historical client status information includes historical CPU occupancy, historical memory occupancy, historical remaining power, historical packet loss rate, historical network delay, and historical bandwidth; A2, candidate client groups are screened out based on the historical client status information and preset status thresholds obtained from the preset database, where the preset status thresholds include preset CPU occupancy thresholds, preset memory occupancy thresholds, remaining power thresholds, preset packet loss rate thresholds, preset network delay thresholds, and bandwidth thresholds, and the candidate client group indicates that the historical CPU occupancy, historical memory occupancy, historical packet loss rate, and historical network delay are not higher than the corresponding thresholds, and the historical A client to be tested whose remaining power and historical bandwidth are not less than the corresponding threshold value; A3, sending request response information to the candidate client group respectively. If no response information is received within the preset time period, the corresponding client to be tested will be deleted from the candidate client group. Otherwise, after the response time record of the client to be tested is completed, the candidate client group is sorted according to the response time. The preset time period should be determined by the preset personnel according to the efficiency requirements of the task to be trained. The response time means the time consumed from sending the request response information to receiving the response information. The sorting means sorting from small to large according to the response time of the client to be tested; A4, judging whether the total number of clients to be tested in the candidate client group is not less than the preset first number of clients. If it is not less than the preset first number of clients, the clients to be tested with the preset first number of clients are extracted from the candidate clients and recorded as the initial client group to be tested. Otherwise, a prompt message of missing clients is sent to the preset personnel. The preset first number of clients is determined by the preset personnel according to the training requirements of the task to be trained.

[0034] In this embodiment, historical client status information refers to the client status information last sent by the corresponding client to be tested and stored in the preset database. The preset first client number is set to twice the minimum client requirement of the task to be trained. The minimum client requirement can be determined by the average of the overall input sample number of the task to be trained and the sample number of the client to be tested. For example, the number of samples of the client to be tested in the historical time period is obtained from the preset database, and the average value is 200, while the overall input sample number of the task to be trained is 5000. Therefore, the minimum client requirement is 5000 / 200=25, and the preset first client number is twice the minimum client requirement, that is, 50.

[0035] The setting of the preset state threshold and the preset time period should be determined based on the mean and standard deviation of historical data. The preset CPU usage threshold, the preset memory usage threshold, the preset packet loss rate threshold, the preset network delay threshold and the preset time period can be expressed by the mean of the corresponding data of the historical time period plus two times the standard deviation. The bandwidth threshold and the remaining power threshold can be expressed by the mean of the corresponding data of the historical time period minus two times the standard deviation. For example, the historical CPU usage mean of a client is 40% and the standard deviation is 10%. The preset CPU usage threshold is the mean plus two times the standard deviation, that is, 60%. This setting method can cover most of the historical state data, ensuring that the selected candidate clients have high stability and availability, thereby improving the efficiency of the distributed scheduling method.

[0036] Furthermore, a matching evaluation is performed based on the client status information, sample information, and task data obtained from a preset database to obtain a resource matching index of the initial client group to be tested, which also includes performing a status evaluation on the client status information of the client to be tested to obtain a real-time status index; the real-time status index is used to describe the current available computing resources and network transmission capabilities of different clients to be tested; the specific process of performing a status evaluation on the client status information of the client to be tested to obtain the real-time status index is as follows: obtaining a status index evaluation weight from a preset database, the status index evaluation weight including a first status weight, a second status weight, a third status weight, a fourth status weight, a fifth status weight, and a sixth status weight, the first status weight is used to describe the CPU occupancy rate to the real-time The second state weight is used to describe the influence of the memory occupancy rate on the real-time state index, the third state weight is used to describe the influence of the remaining power on the real-time state index, the fourth state weight is used to describe the influence of the network delay on the real-time state index, the fifth state weight is used to describe the influence of the packet loss rate on the real-time state index, and the sixth state weight is used to describe the influence of the bandwidth on the real-time state index; the real-time state index of the client to be tested is obtained according to the client state information, the state index evaluation weight and the preset state threshold, the client state information includes computing resource data and network transmission data, the computing resource data includes CPU occupancy rate, memory occupancy rate and remaining power, and the network transmission data includes network delay, packet loss rate and bandwidth.

[0037] The specific calculation formula of the real-time status index is as follows:

[0038] ;

[0039] Where n is the number of the client to be tested in the initial client group to be tested, , N is the preset number of first clients, is the real-time status index of the nth client to be tested in the initial client group to be tested, is the CPU usage of the nth client to be tested in the initial client group to be tested, is the memory usage of the nth client to be tested in the initial client group to be tested, is the remaining power of the nth client to be tested in the initial client group to be tested, is the network delay of the nth client under test in the initial client group under test, is the packet loss rate of the nth client to be tested in the initial client group to be tested, is the bandwidth of the nth client to be tested in the initial client group to be tested, To preset the CPU usage threshold, To preset the memory usage threshold, is the remaining power threshold, To preset the network delay threshold, is the preset packet loss rate threshold, is the bandwidth threshold, is the first state weight, is the second state weight, is the third state weight, is the fourth state weight, is the fifth state weight, is the sixth state weight.

[0040] In this embodiment, the CPU occupancy rate is used to describe the proportion of processor resources currently used by the client to be tested. The larger the CPU occupancy rate, the fewer processor resources are currently remaining in the client to be tested, and therefore the lower the real-time status index is; the memory occupancy rate is used to describe the proportion of memory resources currently used by the client to be tested. The larger the memory occupancy rate, the fewer memory resources are currently remaining in the client to be tested, and therefore the lower the real-time status index is; the remaining power is used to describe the remaining battery power of the client to be tested. The larger the remaining power, the more sufficient the power of the client to be tested, and therefore the higher the real-time status index is; the network delay is used to describe the current network response time of the client to be tested. The larger the network delay, the lower the current network transmission efficiency of the client to be tested, and therefore the lower the real-time status index is; the packet loss rate is used to describe the packet loss ratio of the current network data transmission of the client to be tested. The larger the packet loss rate, the lower the network transmission quality of the client to be tested. The higher the bandwidth, the stronger the network transmission capacity of the client under test, so the higher the real-time status index; the status index evaluation weight is used to adjust the influence of the client status information on the real-time status index. The larger the weight value, the greater the influence of these factors on the real-time status index; the real-time status index is calculated by the client status information, the status index evaluation weight and the preset status threshold, and is used to describe the current available computing resources and network transmission capacity of different clients under test. The larger the real-time status index, the better the current available computing resources and network transmission capacity of the client under test; the real-time status index is obtained by comprehensively evaluating the resource usage and network transmission status of the client under test, which provides a data basis for the subsequent matching evaluation, thereby improving the reliability of the matching evaluation in the distributed scheduling method.

[0041] It should be understood that CPU usage refers to the ratio of CPU resources used by the client under test at the preset time to the total available CPU resources, which is obtained through the cpu_percent() function in the psutil library; memory usage refers to the ratio of used memory to total memory of the client under test at the preset time, which is obtained through the virtual_memory() function in the psutil library; remaining power refers to the ratio of battery power to total battery capacity of the client under test at the preset time, which is obtained through the battery() function in the psutil library; network delay refers to the time delay from the source to the destination of the data packet of the client under test at the preset time, which is obtained through the ping command; packet loss rate refers to the ratio of data packets lost in network transmission of the client under test at the preset time to the total number of data packets, which is obtained by sending a certain number of ping requests and calculating the number of lost requests; bandwidth refers to the amount of data that can be transmitted by the client under test per unit time of the network at the preset time, which is obtained through the speedtest-cli library.

[0042] In order to simplify the independent variable description of the real-time status index, the CPU index is defined , Define the memory index for the CPU index of the nth client to be tested in the initial client group to be tested , Defines the power index for the nth client to be tested in the initial client group. , Define the delay index as the power index of the nth client to be tested in the initial client group to be tested , The packet loss rate index is defined as the delay index of the nth client under test in the initial client group under test. , Define the bandwidth index as the packet loss rate index of the nth client to be tested in the initial client group to be tested , is the bandwidth index of the nth client to be tested in the initial client group to be tested. The simplified formula is: ,by is 0.1, is 0.2, is 0.1, is 0.2, is 0.3, Taking 0.1 as an example, the statistics of the real-time status index change are shown in Table 1:

[0043] Table 1 Statistics of changes in real-time status index:

[0044] ;

[0045] According to the first and second rows of data in Table 1, it can be found that when the CPU index increases, the real-time status index decreases, and the CPU index is negatively correlated with the real-time status index; according to the first and third rows of data, it can be found that when the memory index increases, the real-time status index decreases, and the memory index is negatively correlated with the real-time status index. According to the first and fourth rows of data, it can be found that when the power index increases, the real-time status index increases, and the power index is positively correlated with the real-time status index; according to the first and fifth rows of data, it can be found that when the delay index increases, the real-time status index decreases, and the delay index is negatively correlated with the real-time status index; according to the first and sixth rows of data, it can be found that when the packet loss rate index increases, the real-time status index decreases, and the packet loss rate index is negatively correlated with the real-time status index; according to the first and seventh rows of data, it can be found that when the bandwidth index increases, the real-time status index increases, and the bandwidth index is positively correlated with the real-time status index.

[0046] Specifically, the first state weight is the weight corresponding to the CPU occupancy rate preset in the preset database, which represents the numerical value of the influence degree of the CPU occupancy rate on the real-time state index. When used, the weight corresponding to the preset CPU occupancy rate can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the CPU occupancy rate value of the client to be tested and the weight corresponding to the CPU occupancy rate preset in the preset database form a mapping set, and the real-time CPU occupancy rate value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0047] Specifically, the second state weight is the weight corresponding to the preset memory occupancy rate in the preset database, which represents the numerical value of the influence of the memory occupancy rate on the real-time state index. When used, the weight corresponding to the preset memory occupancy rate can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the memory occupancy rate value of the client to be tested and the weight corresponding to the preset memory occupancy rate in the preset database form a mapping set, and the real-time memory occupancy rate value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0048] Specifically, the third state weight is the weight corresponding to the remaining power preset in the preset database, which represents the value of the influence of the remaining power on the real-time state index. When used, the weight corresponding to the preset remaining power can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the remaining power value of the client to be tested and the weight corresponding to the remaining power preset in the preset database form a mapping set, and the real-time remaining power value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0049] Specifically, the weight corresponding to the preset network delay in the fourth state weight represents the numerical value of the influence of the network delay on the real-time state index. When used, the weight corresponding to the preset network delay can be directly obtained from the preset database. The corresponding relationship can be a preset mapping relationship. For example, the network delay value of the client to be tested and the weight corresponding to the preset network delay in the preset database form a mapping set, and the real-time network delay value is input into the mapping set to obtain the corresponding weight. The mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0050] Specifically, the fifth state weight is the weight corresponding to the packet loss rate preset in the preset database, which represents the numerical value of the influence of the packet loss rate on the real-time state index. When used, the weight corresponding to the preset packet loss rate can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the packet loss rate value of the client to be tested and the weight corresponding to the packet loss rate preset in the preset database form a mapping set, and the real-time packet loss rate value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1]; the sum of the sixth state weight and the first state weight, the second state weight, the third state weight, the fourth state weight and the fifth state weight is 1.

[0051] The algorithm of this embodiment combines "CPU occupancy rate of the nth client to be tested in the initial client group to be tested", "memory occupancy rate of the nth client to be tested in the initial client group to be tested", "remaining power of the nth client to be tested in the initial client group to be tested", "network delay of the nth client to be tested in the initial client group to be tested", "packet loss rate of the nth client to be tested in the initial client group to be tested" and "bandwidth of the nth client to be tested in the initial client group to be tested", and comprehensively analyzes to obtain "real-time status index of the nth client to be tested in the initial client group to be tested", and each parameter is related and does not exist independently. It is a whole, and it is not possible to obtain the "real-time status index of the nth client to be tested in the initial client group to be tested" by simple combination and addition. For example, in this formula, "CPU occupancy rate of the nth client to be tested in the initial client group to be tested" is conducive to increasing "nth client to be tested in the initial client group to be tested". "The real-time status index of the nth client to be tested in the initial client group to be tested" does not necessarily mean that the "real-time status index of the nth client to be tested in the initial client group to be tested" is large. At this time, it is also necessary to consider "the memory occupancy rate of the nth client to be tested in the initial client group to be tested", "the remaining power of the nth client to be tested in the initial client group to be tested", "the network delay of the nth client to be tested in the initial client group to be tested", "the packet loss rate of the nth client to be tested in the initial client group to be tested" and "the bandwidth of the nth client to be tested in the initial client group to be tested" can also determine the current available computing resources and network transmission capacity of the client to be tested. The correlation and mutual influence between various factors are considered in a quantitative way, and the algorithm itself is obtained through comprehensive analysis, which can accurately and quantitatively analyze the current available computing resources and network transmission capacity of the client to be tested, indicating the effect of comprehensive evaluation of the real-time status index. Therefore, accurate analysis is carried out by establishing mathematical forms.

[0052] Furthermore, the specific process of performing matching evaluation according to the client status information, sample information and task data obtained from the preset database to obtain the resource matching index of the initial client group to be tested is as follows: obtaining matching evaluation weights from the preset database, the matching evaluation weights include a first matching weight, a second matching weight and a third matching weight, the first matching weight is used to describe the degree of influence of the number of samples on the resource matching index, the second matching weight is used to describe the degree of influence of the number of sample dimensions on the resource matching index, and the third matching weight is used to describe the degree of influence of the real-time status index on the resource matching index; the resource matching index of the client to be tested is obtained through the real-time status index, sample information, task data and matching evaluation weights, the sample information includes the number of samples and the number of sample dimensions, the number of samples represents the number of data in the local data set used for the task to be trained by different clients to be tested in the initial client group to be tested, the number of sample dimensions represents the average number of features contained in each data of the local data set, the task data includes the model parameter quantity, the number of model layers, the number of input samples and the number of input sample dimensions, and the specific calculation formula of the resource matching index is as follows:

[0053] ;

[0054] In the formula, is the resource matching index of the nth client to be tested in the initial client group to be tested, is the number of samples of the nth client to be tested in the initial client group to be tested, is the number of sample dimensions of the nth client to be tested in the initial client group to be tested, is the number of input samples of the task to be trained, is the number of dimensions of the input samples of the task to be trained, is the model parameter of the task to be trained, is the number of model layers of the task to be trained, is the first matching weight, is the second matching weight, is the third matching weight, is the complexity adjustment factor.

[0055] In this embodiment, the number of samples is used to describe the amount of data in the local data set of the client to be tested. The larger the number of samples, the richer the local data of the client to be tested, and therefore the higher the resource matching index; when the number of samples is lower than the input number of samples, the resource matching index is 0; the number of sample dimensions is used to describe the average number of features contained in each piece of data in the local data set of the client to be tested. The larger the number of sample dimensions, the richer the feature information of the local data, and therefore the higher the resource matching index; the real-time status index is used to describe the currently available computing resources and network transmission capabilities of different clients to be tested, the number of model parameters and the number of model layers are used to describe the complexity of the task to be trained, and the ratio of the two is used to describe the degree to which the computing resources and network transmission capabilities of the client to be tested meet the training task. The complexity adjustment factor is used to describe the influence of the model parameter quantity and the number of model layers on the ratio value. The larger the ratio, the higher the resource matching index; the matching evaluation weight is used to adjust the influence of the number of samples, the number of sample dimensions and the real-time status index on the resource matching index. The larger the weight value, the greater the influence of these factors on the resource matching index; the resource matching index obtained by multiple parameters is used to describe the degree of satisfaction of the computing resources, network transmission capabilities and data sets of different clients to be tested for the training tasks. The higher the resource matching index, the more the computing resources, network transmission capabilities and data sets of the client to be tested meet the requirements of the training tasks, which provides a data basis for the subsequent screening and task allocation of the initial clients to be tested, thereby improving the reliability of the distributed scheduling method.

[0056] It should be understood that the number of samples is obtained by counting Python's len function, and the number of sample dimensions is obtained using the average value of NumPy's data.shape. The number of model parameters refers to the total number of learnable parameters in the training task, which is usually related to the complexity and expressiveness of the model. It is obtained by manually calculating the number of parameters of each layer, or it can be automatically obtained using the odel.summary function in the TensorFlow framework. The number of model layers refers to the number of different types of layers contained in the model, including input layers, hidden layers, and output layers. The number of layers can be manually counted by checking the model definition code, or it can be directly obtained using the model.summary function in Keras. The number of input samples and the number of input sample dimensions are manually set according to task requirements. The number of input samples refers to the minimum number of data samples used for training, verification, or testing; the number of input sample dimensions refers to the minimum number of features of samples in the sample set.

[0057] In order to simplify the description of the independent variable of the resource matching index, the sample number index is defined as , The sample dimension index is defined as the sample number index of the nth client to be tested in the initial client group to be tested. , Define the resource index as the sample dimension index of the nth client to be tested in the initial client group to be tested , is the resource index of the nth client to be tested in the initial client group to be tested, and since Can be simplified to , the simplified formula is ,by is 0.2, is 0.3, Taking 0.5 as an example, Figure 2 As shown, it is a variable relationship diagram of the resource matching index provided in this embodiment. It can be seen from the figure that when the sample number index is less than 1, the resource matching index is 0. When the sample number index is not less than 1, the sample number index, sample dimension index and resource index contribute more to the growth of the resource matching index when the resource matching index is at a low value. As the variable value increases, the growth of the resource matching index gradually slows down and finally reaches a stable state.

[0058] Specifically, the first matching weight is the weight corresponding to the preset number of samples in the preset database, which represents the numerical value of the influence of the number of samples on the resource matching index. When used, the weight corresponding to the preset number of samples can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the sample number value of the client to be tested and the weight corresponding to the preset number of samples in the preset database form a mapping set, and the real-time sample number value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0059] Specifically, the second matching weight is the weight corresponding to the preset sample dimension number in the preset database, which represents the numerical value of the influence of the sample dimension number on the resource matching index. When used, the weight corresponding to the preset sample dimension number can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the sample dimension number value of the client to be tested and the weight corresponding to the preset sample dimension number in the preset database form a mapping set, and the real-time sample dimension number value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1]; the sum of the third matching weight, the first matching weight, and the second matching weight is 1.

[0060] Specifically, the complexity adjustment factor is the weight corresponding to the model parameter quantity and the number of model layers preset in the preset database, which represents the numerical value of the influence of the model parameter quantity and the number of model layers on the resource matching index. When used, the weight corresponding to the sum of the preset model parameter quantity and the number of model layers can be directly obtained from the preset database. The corresponding relationship can be a pre-set mapping relationship. For example, the sample number value of the client to be tested and the weight corresponding to the sum of the model parameter quantity and the number of model layers preset in the preset database form a mapping set, and the real-time model parameter quantity and the sum of the number of model layers are input into the mapping set to obtain the corresponding weight. The mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0061] The algorithm of this embodiment combines "the number of samples of the nth client to be tested in the initial client group to be tested", "the number of sample dimensions of the nth client to be tested in the initial client group to be tested" and "the real-time status index of the nth client to be tested in the initial client group to be tested", and comprehensively analyzes to obtain the "resource matching index of the nth client to be tested in the initial client group to be tested", and each parameter is related to each other and does not exist independently. It is a whole, and the "resource matching index of the nth client to be tested in the initial client group to be tested" cannot be obtained by simple combination and addition. For example, in this formula, "the number of samples of the nth client to be tested in the initial client group to be tested" is conducive to increasing the "resource matching index of the nth client to be tested in the initial client group to be tested". , does not necessarily mean that the "resource matching index of the nth client to be tested in the initial client group to be tested" is large. At this time, it is also necessary to consider the "number of sample dimensions of the nth client to be tested in the initial client group to be tested" and the "real-time status index of the nth client to be tested in the initial client group to be tested". It can also determine the computing resources, network transmission capacity and data set satisfaction degree of the client to be tested for the training task. The correlation and mutual influence between various factors are considered in a quantitative way. The algorithm itself is obtained through comprehensive analysis, which can accurately and quantitatively analyze the computing resources, network transmission capacity and data set satisfaction degree of the client to be tested for the training task, indicating the effect of comprehensive evaluation of the resource matching index. Therefore, accurate analysis is carried out by establishing mathematical form.

[0062] Furthermore, the specific steps of screening the initial client group to be tested according to the resource matching index to obtain the client group to be tested are as follows: B1, obtaining a preset matching threshold from a preset database, which should be preset by a preset person according to the expected task data of the task to be trained; B2, traversing the initial client group to be tested, and judging whether the resource matching index of the client to be tested is not lower than the preset matching threshold, if it is not lower than the preset matching threshold, the corresponding client to be tested is recorded in the client group to be selected, otherwise no processing is performed; B3, sorting the client group to be selected in order according to the resource matching index; B4, judging whether the total number of clients to be tested in the client group to be selected is not less than a preset second number of clients, if it is not less than the preset second number of clients, extracting the clients to be tested of the preset second number of clients from the client group to be selected and recording them as the client group to be tested, otherwise sending a prompt message of missing clients to the preset person, and the preset second number of clients is determined by the preset person according to the training requirements of the task to be trained.

[0063] In this embodiment, the setting of the preset matching threshold should be determined based on the resource matching index of the historical time period. For example, by obtaining the resource matching index of the historical time period from a preset database and performing statistics to obtain the mean and standard deviation, the preset matching threshold is set to the mean of the resource matching index plus twice the standard deviation; the ordered sorting here refers to sorting from large to small according to the resource matching index of the client to be tested; the number of second clients is preset to half of the number of first clients, and a group of clients to be tested that meet the requirements of the task to be trained is obtained through screening, thereby further improving the reliability of the distributed scheduling method.

[0064] Furthermore, the specific steps of task allocation are as follows: C1, the resource matching index of the client group to be tested is counted to obtain the total resource matching index, and the resource weight of the corresponding client to be tested is obtained in combination with the resource matching index. The total resource matching index represents the sum of the resource matching index of all the clients to be tested in the client group to be tested, and the resource weight represents the ratio of the resource matching index of the client to be tested to the total resource matching index; C2, the nth round of training tasks is counted to obtain the total number of task blocks and task block numbers of the nth round, and the number of task blocks corresponding to the client to be tested is obtained in combination with the resource weight. The nth round of training tasks is determined by the preset personnel according to the specific task volume of the tasks to be trained. The total number of task blocks of the nth round of training tasks represents the total number of task blocks of the nth round of training tasks in the tasks to be trained, and the task block number represents the number of task blocks of the training tasks to be trained. The task blocks of the nth round of training tasks are numbered, and the task block represents the minimum computing unit of the task to be trained. The number of task blocks represents the product of the total number of task blocks in the nth round and the resource weight; C3, traverse the client group to be tested, and determine whether the number of task blocks corresponding to the client to be tested is greater than the preset task block threshold. If it is greater than the preset task block threshold, the task block number of the preset task block threshold is stored in the task queue, otherwise the task block number of the task block number is stored in the task queue. The task queue represents a set of task block numbers arranged from most to least according to the number of task blocks. The preset task block threshold is determined by the preset personnel according to the training efficiency of the distributed scheduling method; C4, determine whether there is a task block in the nth round of training tasks that is not written into the task queue. If so, the task block is merged into the n+1th round of training tasks, otherwise it is not processed.

[0065] In this embodiment, resource weights are calculated by using a resource matching index to ensure that clients with high resource matching degrees obtain more tasks. This allocation method can make full use of the resources of high-performance clients, thereby improving the overall task execution efficiency. The preset task block threshold can be determined by the total number of task blocks in the nth round and the preset number of second clients. For example, twice the ratio of the total number of task blocks in the nth round to the preset number of second clients is set as the preset task block threshold. The preset task block threshold is used to limit the number of task blocks allocated to a single client to be tested, thereby ensuring that the task load of each client to be tested is not too much, thereby improving the balance of task allocation in the distributed scheduling method.

[0066] Furthermore, the specific steps of performing transmission analysis on the client status information to obtain the transmission strategy of the client group to be tested are as follows: D1, judging whether the bandwidth of the client to be tested is not greater than the bandwidth selection threshold, if it is not greater than the bandwidth selection threshold, selecting the UDP protocol as the transmission protocol, otherwise selecting the TCP protocol as the transmission protocol, and the transmission protocol represents a set of rules for transmitting data to the predicted client; D2, judging whether the network delay of the client to be tested is not greater than the delay selection threshold, if it is not greater than the delay selection threshold, selecting no compression as the compression strategy, otherwise selecting lossy compression as the compression strategy, and the compression strategy represents a method of compressing the task block to reduce its transmission data volume; D3, judging whether the packet loss rate of the client to be tested is not greater than the packet loss rate selection threshold, if it is not greater than the packet loss rate selection threshold, selecting a preset first number as the transmission number, otherwise selecting a preset second number as the transmission number, and the transmission number represents the number of task blocks sent in each transmission cycle.

[0067] In this embodiment, the UDP protocol (User Datagram Protocol) is a connectionless transport layer protocol, which is suitable for situations with low bandwidth and can transmit data faster; the TCP protocol (Transmission Control Protocol) is a connection-oriented transport layer protocol, which is suitable for situations where the bandwidth of the client to be tested is sufficient, and can ensure the reliable transmission and integrity of the task queue; lossless compression (LosslessCompression) is a data compression method, which can not lose any information during the compression and decompression process, and is suitable for situations where the network delay of the client to be tested is low, which can reduce the data volume and ensure the complete transmission of the task queue; lossy compression (Lossy Compression) is a data compression method, which reduces the data volume by discarding some unimportant data, and is suitable for situations where the network delay of the client to be tested is high, and can speed up the transmission speed of the task queue, but some data may not be restored.

[0068] The bandwidth selection threshold is represented by the average bandwidth of the client to be tested in the historical time period, the delay selection threshold is represented by the average delay time of the client to be tested in the historical time period, and the packet loss rate selection threshold is represented by the average packet loss rate of the client to be tested in the historical time period; the preset first number represents the maximum number of task blocks that the client to be tested can receive in each transmission cycle. To determine the maximum transmission capacity of the client, it is necessary to measure the maximum transmission capacity of the client under different network conditions through experiments. The experiment is as follows: gradually increase the number of task blocks in each transmission cycle until the packet loss rate of the transmission exceeds the preset packet loss rate threshold in the preset state threshold. The number of task blocks at this time is the maximum transmission capacity of the client, that is, the preset first number; the preset second number represents the number of task blocks received by the client in each transmission cycle under the condition of high packet loss rate. In this embodiment, it is set to half of the preset first number to ensure that when the packet loss rate is greater than the packet loss rate selection threshold, the client will not cause transmission failure or excessive packet loss due to excessive load; through dynamic judgment and adjustment of bandwidth, delay and packet loss rate, the optimal transmission protocol, compression strategy and task block transmission number are selected, thereby improving the efficiency and reliability of data transmission in the distributed scheduling method.

[0069] Furthermore, the specific process of performing task evaluation based on task execution data to obtain the task execution evaluation index of the client group to be tested is as follows: obtaining task evaluation weights from a preset database, the task evaluation weights including a first task execution weight, a second task execution weight and a third task execution weight, the first task execution weight being used to describe the degree of influence of the model error rate on the resource task evaluation index, the second task execution weight being used to describe the degree of influence of the training loss value on the resource task evaluation index, and the third task execution weight being used to describe the degree of influence of the task completion time on the resource task evaluation index; obtaining expected task data from a preset database, the expected task data including a model error rate reference value, a training loss value reference value and a task completion time reference value; obtaining the task execution evaluation index of the client to be tested through the task execution data, the expected task data and the task evaluation weights, the task execution data including the model error rate, the training loss value and the task completion time.

[0070] Among them, the specific calculation formula of the task execution evaluation index is as follows:

[0071] ;

[0072] Where m is the number of the client to be tested in the client group to be tested, , M is the preset number of second clients, is the task execution evaluation index of the mth client under test in the client group under test, is the model error rate of the mth client to be tested in the client group to be tested, is the training loss value of the mth client to be tested in the client group to be tested, is the task completion time of the mth client under test in the client group under test, is the reference value of the model error rate, is the reference value of training loss, is the reference value of task completion time, is the first task execution weight, Execution weights for the second task, Perform weighting for the third task.

[0073] In this embodiment, the model error rate reference value is represented by the average of the model error rate in the historical time period, the training loss value reference value is represented by the average of the training loss values ​​in the historical time period, and the task completion time reference value is represented by the average of the task completion time in the historical time period; the model error rate is used to describe the proportion of prediction errors in the task execution process of the client under test. The larger the model error rate, the worse the task execution quality, and therefore the lower the task execution evaluation index; the training loss value is used to describe the error of the model in the training process of the client under test. The larger the training loss value, the greater the deviation between the model's prediction result and the actual result, and therefore the lower the task execution evaluation index; the task completion time is used to describe the time required for the client under test to complete the task queue. The longer the task completion time is, the lower the task execution efficiency is, and therefore the lower the task execution evaluation index is. The task execution weight is used to adjust the influence of the model error rate, training loss value and task completion time on the task execution evaluation index. The larger the weight value is, the greater the influence of these factors on the task execution evaluation index is. The task execution evaluation index is used to describe the comprehensive performance of the client under test when executing the task. The higher the task execution evaluation index is, the better the comprehensive performance of the client under test when executing the task is. The task execution evaluation index obtained by the model error rate, training loss value, task completion time and task evaluation weight provides an effective evaluation basis for subsequent dynamic optimization and improves the reliability of dynamic optimization in the distributed scheduling method.

[0074] It should be understood that the model error rate refers to the proportion of samples with model prediction errors after the training of the client under test is completed, which is obtained by the accuracy_score function in the sklearn.metrics library in Python. The training loss value refers to the difference between the prediction of the model on the training data after training and the actual label, which is obtained by the binary_crossentrop function of TensorFlow. The task completion time refers to the time from the client under test receiving the task block to the client under test completing the task block.

[0075] Specifically, the first task execution weight is the weight corresponding to the preset model error rate in the preset database, which represents the numerical value of the influence of the model error rate on the task execution evaluation index. When used, the weight corresponding to the preset model error rate can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the model error rate value of the client to be tested and the weight corresponding to the preset model error rate in the preset database form a mapping set, and the real-time model error rate value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1];

[0076] Specifically, the second task execution weight is the weight corresponding to the preset training loss value in the preset database, which represents the numerical value of the influence of the training loss value on the task execution evaluation index. When used, the weight corresponding to the preset training loss value can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the training loss value of the client to be tested and the weight corresponding to the preset training loss value in the preset database form a mapping set, and the real-time training loss value is input into the mapping set to obtain the corresponding weight, wherein the mapping relationship can be one-to-one or many-to-one. In this example, its value range is [0, 1]; the sum of the third task execution weight, the first task execution weight, and the second task execution weight is 1.

[0077] The algorithm of this embodiment combines the "model error rate of the mth client to be tested in the client group to be tested", "the training loss value of the mth client to be tested in the client group to be tested" and "the task completion time of the mth client to be tested in the client group to be tested", and comprehensively analyzes to obtain the "task execution evaluation index of the mth client to be tested in the client group to be tested", and each parameter is related to each other and does not exist independently. It is a whole, and the "task execution evaluation index of the mth client to be tested in the client group to be tested" cannot be obtained by simple combination and addition. For example, in this formula, the "model error rate of the mth client to be tested in the client group to be tested" is conducive to increasing the "task execution evaluation index of the mth client to be tested in the client group to be tested". The "task execution evaluation index of the client to be tested" does not necessarily mean that the "task execution evaluation index of the mth client to be tested in the client group to be tested" is large. At this time, it is also necessary to consider the "training loss value of the mth client to be tested in the client group to be tested" and the "task completion time of the mth client to be tested in the client group to be tested". It can also determine the comprehensive performance of the client to be tested in executing the task. The correlation and mutual influence between various factors are considered in a quantitative way. The algorithm itself is obtained through comprehensive analysis, which can accurately and quantitatively analyze the comprehensive performance of the client to be tested in executing the task, indicating the effect of comprehensive evaluation of the task execution evaluation index. Therefore, precise analysis is carried out by establishing mathematical forms.

[0078] Furthermore, the specific process of dynamic optimization based on the task execution evaluation index and the preset task evaluation threshold is as follows: traverse the client group to be tested, determine whether the task execution evaluation index of the client to be tested is not less than the preset task evaluation threshold, if it is not less than the preset task evaluation threshold, do not process it, otherwise mark the corresponding task block as an unfinished task block, and recalculate the resource matching index of the corresponding client to be tested; after the traversal is completed, retrain the unfinished task block, sort the client group to be tested in order according to the resource matching index, and perform task allocation, retraining means sending the unfinished task block to the adjacent client to be tested for re-execution.

[0079] In this embodiment, the task blocks and clients to be tested whose task execution evaluation index is less than a preset task evaluation threshold are found by traversing the clients to be tested; the task blocks are marked as incomplete, and retraining is performed to ensure that the task blocks are eventually completed; the resource matching index of the clients to be tested is recalculated to evaluate the degree to which the computing resources, network transmission capacity and data set of the clients to be tested meet the training tasks, and the next round of task allocation is performed after orderly sorting; the orderly sorting here refers to sorting from large to small according to the resource matching index of the clients to be tested; the high quality and high efficiency of the execution of the tasks to be trained are ensured through dynamic optimization, and the resource utilization rate of the distributed scheduling method and the flexibility of task allocation are enhanced.

[0080] Furthermore, dynamic optimization is performed according to the task execution evaluation index and the preset task evaluation threshold until the training of the task to be trained is completed. After that, client visualization is also included. The client visualization represents the status of the client to be tested, the resource matching status and the task execution performance through charts. The specific process is as follows: a real-time status index bar chart is drawn according to the changing relationship between the real-time status index and the number of the client to be tested in the initial client group to be tested; a resource matching index bar chart is drawn according to the changing relationship between the resource matching index and the number of the client to be tested in the initial client group to be tested; a task evaluation scatter plot is drawn according to the relationship between the task execution evaluation index and the number of the client to be tested in the client group to be tested.

[0081] In this embodiment, the client visualization chart can be used to intuitively display the status of the client to be tested, the resource matching status and the task execution performance. Specifically, the horizontal axis of the real-time status index bar chart represents the number of each client to be tested in the initial client group to be tested, and the vertical axis represents the real-time status index, which can help observe and analyze the current operating status of each client and its resource occupancy. The horizontal axis of the resource matching index bar chart represents the client number to be tested, and the vertical axis represents the resource matching index, which intuitively displays the resource adaptability of each client to the current task, which is convenient for the preset personnel to judge the rationality of resource allocation. The horizontal axis of the task execution evaluation index scatter plot represents the client number to be tested, and the vertical axis represents the task execution evaluation index. The task execution effect of each client can be displayed through this chart, which is convenient for discovering the client to be tested whose task execution evaluation index is less than the preset task evaluation threshold and adjusting the strategy in time; these charts provide visual support for the subsequent analysis and decision-making of the preset personnel, and improve the management efficiency of the distributed scheduling method and the accuracy of task allocation.

[0082] To summarize, the embodiment of the present application performs a matching evaluation on the client status information, sample information and task data, screens out the client group to be tested according to the obtained resource matching index, performs task allocation to obtain a task queue, and then transmits the task queue to the client group to be tested according to the transmission strategy. Finally, dynamic optimization is performed based on the task execution evaluation index, thereby improving the execution efficiency of the task to be trained, thereby achieving more stable distributed resource scheduling, and effectively solving the problem of low scheduling stability in the distributed scheduling process based on split federated learning in the prior art.

[0083] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0087] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0088] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A distributed scheduling method based on split federated learning, characterized in that: The following steps are involved: S1, obtaining sample information and client status information of the initial client group to be tested; S2, performing matching evaluation according to the client status information, sample information and task data obtained from a preset database to obtain a resource matching index of the initial client group to be tested, and screening the initial client group to be tested according to the resource matching index to obtain a client group to be tested, wherein the resource matching index is used to describe the degree to which the computing resources, network transmission capabilities and data sets of different clients to be tested satisfy the training task; S3, assigning the training tasks to be tested to obtain a task queue according to the resource matching index of the client group to be tested, and performing transmission analysis on the client status information to obtain a transmission strategy for the client group to be tested, and transmitting the task queue to the client group to be tested according to the transmission strategy; S4, receiving task execution data sent by the client group to be tested, and performing task evaluation according to the task execution data to obtain a task execution evaluation index of the client group to be tested, and performing dynamic optimization according to the task execution evaluation index and a preset task evaluation threshold until the training of the task to be trained is completed, wherein the task execution evaluation index is used to quantitatively describe the comprehensive performance of the client to be tested when executing the task; The obtaining of sample information and client status information of the initial client group to be tested also includes filtering the initial client group to be tested from the client pool, and the specific steps are as follows: A1, obtaining historical client status information of all clients to be tested in the client pool from a preset database, wherein the historical client status information includes historical CPU occupancy, historical memory occupancy, historical remaining power, historical packet loss rate, historical network delay and historical bandwidth; A2, selecting a candidate client group according to historical client status information and a preset status threshold obtained from a preset database; A3, sending request response information to the candidate client group respectively. If no response information is received within a preset time period, the corresponding client to be tested is deleted from the candidate client group. Otherwise, after the response time of the client to be tested is recorded, the candidate client group is sorted according to the response time; A4, determining whether the total number of the clients to be tested in the candidate client group is not less than the preset first number of clients, if not less than the preset first number of clients, extracting the preset first number of clients to be tested from the candidate clients and recording them as the initial client group to be tested, otherwise sending a prompt message of missing clients to a preset person; The matching degree evaluation is performed according to the client status information, the sample information and the task data obtained from the preset database to obtain the resource matching degree index of the initial client group to be tested, and the state evaluation of the client status information of the client to be tested is also performed to obtain the real-time state index; The real-time status index is used to describe the current available computing resources and network transmission capabilities of different clients to be tested; The specific process of performing status evaluation on the client status information of the client to be tested to obtain the real-time status index is as follows: Acquire a state index evaluation weight from a preset database, wherein the state index evaluation weight includes a first state weight, a second state weight, a third state weight, a fourth state weight, a fifth state weight, and a sixth state weight; The real-time status index of the client to be tested is obtained according to the client status information, the status index evaluation weight and the preset status threshold, wherein the client status information includes computing resource data and network transmission data, the computing resource data includes CPU occupancy rate, memory occupancy rate and remaining power, and the network transmission data includes network delay, packet loss rate and bandwidth; The specific process of performing matching evaluation based on the client status information, sample information and task data obtained from the preset database to obtain the resource matching index of the initial client group to be tested is as follows: Acquire matching evaluation weights from a preset database, wherein the matching evaluation weights include a first matching weight, a second matching weight, and a third matching weight; The resource matching index of the client to be tested is obtained through the real-time status index, sample information, task data and matching evaluation weight. The sample information includes the number of samples and the number of sample dimensions. The task data includes the model parameter quantity, the number of model layers, the number of input samples and the number of input sample dimensions. The specific calculation formula of the resource matching index is as follows: ； Where n is the number of the client to be tested in the initial client group to be tested, , N is the preset number of first clients, is the real-time status index of the nth client to be tested in the initial client group to be tested, is the resource matching index of the nth client to be tested in the initial client group to be tested, is the number of samples of the nth client to be tested in the initial client group to be tested, is the number of sample dimensions of the nth client to be tested in the initial client group to be tested, is the number of input samples of the task to be trained, is the number of dimensions of the input samples of the task to be trained, is the model parameter of the task to be trained, is the number of model layers of the task to be trained, is the first matching weight, is the second matching weight, is the third matching weight, is the complexity adjustment factor.

2. A distributed scheduling method based on split federated learning as claimed in claim 1, characterized in that: The specific steps of screening the initial client group to be tested according to the resource matching index to obtain the client group to be tested are as follows: B1, obtaining a preset matching degree threshold from a preset database; B2, traverse the initial client group to be tested, determine whether the resource matching index of the client to be tested is not lower than the preset matching threshold, if it is not lower than the preset matching threshold, record the corresponding client to be tested into the client group to be selected, otherwise do not process it; B3, sorting the client groups to be selected in order according to the resource matching index; B4, judging whether the total number of the clients to be tested in the client group to be selected is not less than the preset second number of clients, if it is not less than the preset second number of clients, extracting the preset second number of clients to be tested from the client group to be selected and recording it as the client group to be tested, otherwise sending a prompt message of missing clients to the preset personnel.

3. A distributed scheduling method based on split federated learning as claimed in claim 1, characterized in that: The specific steps of the task allocation are as follows: C1, the resource matching index of the client group to be tested is counted to obtain the total resource matching index, and the resource weight of the corresponding client to be tested is obtained by combining the resource matching index, wherein the total resource matching index represents the sum of the resource matching indexes of all the clients to be tested in the client group to be tested, and the resource weight represents the ratio of the resource matching index of the client to be tested to the total resource matching index; C2, count the nth round of training tasks to obtain the total number of task blocks and task block numbers of the nth round, and combine the resource weight to obtain the number of task blocks corresponding to the client to be tested; C3, traverse the client group to be tested, determine whether the number of task blocks corresponding to the client to be tested is greater than the preset task block threshold, if it is greater than the preset task block threshold, write the task block number of the preset task block threshold into the task queue, otherwise store the task block number of the task block number into the task queue, the preset task block threshold is determined by the preset personnel according to the training efficiency of the distributed scheduling method; C4, determine whether there is a task block in the nth round of training tasks that is not stored in the task queue. If so, merge the task block into the n+1th round of training tasks, otherwise do not process it.

4. A distributed scheduling method based on split federated learning as claimed in claim 1, characterized in that: The specific steps of performing transmission analysis on the client status information to obtain the transmission strategy of the client group to be tested are as follows: D1, judging whether the bandwidth of the client to be tested is not greater than the bandwidth selection threshold, if it is not greater than the bandwidth selection threshold, selecting the UDP protocol as the transmission protocol, otherwise selecting the TCP protocol as the transmission protocol; D2, determine whether the network delay of the client to be tested is not greater than the delay selection threshold. If it is not greater than the delay selection threshold, select no compression as the compression strategy, otherwise select lossy compression as the compression strategy; D3, determining whether the packet loss rate of the client to be tested is not greater than the packet loss rate selection threshold, if it is not greater than the packet loss rate selection threshold, selecting a preset first number as the transmission number, otherwise selecting a preset second number as the transmission number.

5. A distributed scheduling method based on split federated learning as claimed in claim 1, characterized in that: The specific process of performing task evaluation according to the task execution data to obtain the task execution evaluation index of the client group to be tested is as follows: Acquire task evaluation weights from a preset database, wherein the task evaluation weights include a first task execution weight, a second task execution weight, and a third task execution weight; Acquire expected task data from a preset database, wherein the expected task data includes a model error rate reference value, a training loss value reference value, and a task completion time reference value; The task execution evaluation index of the client to be tested is obtained through task execution data, expected task data and task evaluation weight, and the task execution data includes model error rate, training loss value and task completion time.

6. A distributed scheduling method based on split federated learning as claimed in claim 3, characterized in that: The specific process of dynamic optimization based on the task execution evaluation index and the preset task evaluation threshold is as follows: Traverse the client group to be tested, determine whether the task execution evaluation index of the client to be tested is not less than the preset task evaluation threshold, if not less than the preset task evaluation threshold, do not process it, otherwise mark the corresponding task block as an unfinished task block, and recalculate the resource matching index of the corresponding client to be tested; After the traversal is completed, the unfinished task blocks are retrained, the client groups to be tested are sorted in order according to the resource matching index, and tasks are assigned.

7. A distributed scheduling method based on split federated learning as claimed in claim 1, characterized in that: The dynamic optimization is performed according to the task execution evaluation index and the preset task evaluation threshold until the training task is completed, and then the client visualization is performed. The specific process is as follows: Draw a real-time status index bar chart according to the change relationship between the real-time status index and the number of the client to be tested in the initial client group to be tested; Draw a resource matching index bar chart according to the relationship between the resource matching index and the number of the client to be tested in the initial client group to be tested; A task evaluation scatter plot is drawn based on the relationship between the task execution evaluation index and the number of the client to be tested in the client group to be tested.

Images