Hybrid architecture system and task scheduling method based on data transmission time consumption

Abstract

The present application discloses a hybrid architecture system and a task scheduling method based on data transmission time consumption, the method comprising: submitting a task to be processed, determining the first data associated with the task to be processed, and calculating the first time consumption of transmitting the subset of the first data that has not been uploaded to a third party; receiving the second time consumption of transmitting the subset of the second data that has not been uploaded corresponding to each other second participant to the third party; determining the task type of the task to be processed based on the first time consumption, the second time consumption and a preset time consumption threshold, wherein the task type includes distributed and centralized; when the task to be processed is centralized, sending the subset of the first data that has not been uploaded to the third party, so that the third party can combine the first data and the second data to perform centralized calculations to obtain processing results; when the task to be processed is decentralized, local task scheduling is used to cooperate with decentralized calculations to obtain processing results. The present application selects the least time-consuming calculation method to avoid waste.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a hybrid architecture system and a task scheduling method based on time-consuming data transmission. Background Art

[0002] With the development of artificial intelligence technology, the application of privacy computing (based on federated learning) is becoming more and more extensive. Privacy computing refers to a technology and system for joint computing by two or more participants (the participants in privacy computing can be different departments of the same organization or different organizations, realizing cross-domain cooperation of multi-source data while protecting data security). The participants conduct joint machine learning and joint analysis on their data through collaboration without leaking their respective data.

[0003] During the privacy computing process, different computing methods can be selected. Different computing methods have different data transmission and data storage capacities. In the prior art, users manually select specific computing methods. However, the manually selected computing method is often not the most optimal, resulting in a waste of privacy computing resources. Summary of the invention

[0004] The main purpose of this application is to provide a hybrid architecture system and a task scheduling method based on time-consuming data transmission, aiming to solve the technical problem of manually selecting the calculation method in the existing privacy computing process, which easily leads to waste of privacy computing resources.

[0005] To achieve the above objectives, the present application provides a hybrid architecture system, the hybrid architecture system comprising:

[0006] A first participant, the first participant is used to: initiate a task to be processed, determine first data associated with the task to be processed, and calculate a first time required to transmit a subset of the first data that has not been uploaded to a third party to the third party; also used to receive a second time required to transmit a subset of the second data associated with the task to be processed that has not been uploaded to a third party from each other second participant, and determine a task type of the task to be processed based on the first time, the second time and a preset time threshold, wherein the task type includes a decentralized type and a centralized type; also used to send the subset of the first data that has not been uploaded to a third party to the third party when the task to be processed is of a centralized type, or to coordinate with the third party and other participants to perform decentralized computing through local task scheduling when the task to be processed is of a decentralized type;

[0007] The second participant is used to: after detecting that the first participant has initiated a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant; and also to send the subset of the second data that has not been uploaded to the third party to the third party when the task to be processed is of a centralized type, or to coordinate the third party and other participants to perform distributed computing through local task scheduling when the task to be processed is of a decentralized type;

[0008] A third party, the third party is used to: when the task to be processed is of a centralized type, combine the first data and the second data to perform centralized computing, or when the task to be processed is of a decentralized type, cooperate with other participants to perform decentralized computing through local computing.

[0009] Optionally, there are multiple second participants.

[0010] Optionally, the first participant includes a first scheduling module, a first computing module, a first communication module and a task type recommendation module, the first communication module is used to communicate with the second participant and / or the third party; the task type recommendation module is used to initiate a task to be processed, determine the first data associated with the task to be processed, and calculate the first time required to transmit the subset of the first data that has not been uploaded to the third party to the third party, and is used to receive the second time from each other second participant, and determine the task type of the task to be processed based on the first time, the second time and a preset time threshold; the first scheduling module is used to schedule the first computing module to perform local calculations through local tasks when the task to be processed is a distributed type to cooperate with the third party and other participants to perform distributed calculations, or when the task to be processed is a centralized type, send the subset of the first data that has not been uploaded to the third party to the third party through the first communication module.

[0011] Optionally, the second participant includes a second scheduling module, a second computing module and a second communication module, the second communication module is used to communicate with the first participant and / or the third party, the second scheduling module is used to: after detecting that the first participant initiates a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant through the second communication module, and when the task to be processed is a distributed type, schedule the second computing module through local tasks to cooperate with the third party and other participants to perform distributed computing, or when the task to be processed is a centralized type, send the subset of the second data that has not been uploaded to the third party to the third party through the second communication module.

[0012] Optionally, the third party includes a third scheduling module, a third computing module and a third communication module, and the third communication module is used to communicate with the first participant and / or the second participant; the third scheduling module is used to: when the task to be processed is of a centralized type, schedule the third computing module to perform centralized calculations in combination with the first data and the second data, or when the task to be processed is of a distributed type, schedule the third computing module to perform local calculations through local tasks to cooperate with other participants to perform distributed calculations.

[0013] To achieve the above-mentioned object, the present application provides a task scheduling method based on data transmission time consumption, which is applied to any first participant in a participant subsystem that initiates a task to be processed, wherein the participant subsystem also includes at least one second participant associated with the task to be processed, and the task scheduling method includes:

[0014] Submitting a task to be processed, determining first data associated with the task to be processed, and calculating a first time required to transmit a subset of the first data that has not been uploaded to a third party to the third party;

[0015] receiving a second time required for transmitting a subset of the second data associated with the to-be-processed task from each other second participant and not uploaded to the third party to the third party;

[0016] Determining a task type of the to-be-processed task based on the first time consumption, the second time consumption, and a preset time consumption threshold, wherein the task type includes a decentralized type and a centralized type;

[0017] When the task to be processed is of a centralized type, a subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party after the second participant receives the task type to perform centralized calculations to obtain processing results; or when the task to be processed is of a distributed type, local task scheduling is used to cooperate with the third party and other participants to perform distributed calculations to obtain processing results.

[0018] Optionally, the step of determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold comprises:

[0019] Selecting a maximum time from the first time and the second time as a first data transmission time;

[0020] If the first data transmission time is greater than a preset time threshold, determining that the task type of the task to be processed is a distributed type;

[0021] If the first data transmission time is less than or equal to a preset time threshold, it is determined that the task type of the to-be-processed task is a centralized type.

[0022] Optionally, the step of determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold comprises:

[0023] Obtaining task information of the task to be processed;

[0024] Based on the task information, query from preset historical tasks a first historical task that matches the similarity of the task to be processed and whose task type is a decentralized type;

[0025] Obtaining the intermediate transmission volume during the process of transmitting intermediate data between the first participant and other second participants in the process of processing the first historical task, and calculating the second data transmission time based on the intermediate transmission volume and current network status information;

[0026] Selecting a maximum time from the first time and the second time as a first data transmission time;

[0027] If the first data transmission time is greater than or equal to the second data transmission time, determining that the type of the task to be processed is a distributed type;

[0028] If the first data transmission time is less than the second data transmission time, it is determined that the type of the task to be processed is a centralized type.

[0029] To achieve the above-mentioned purpose, the present application also provides a task scheduling method based on data transmission time consumption, which is applied to any second participant in a participant subsystem that is passively associated with a task to be processed, wherein the participant subsystem also includes a first participant that initiates the task to be processed, and the task scheduling method includes:

[0030] After detecting that the first participant has initiated a task to be processed, determining second data associated with the task to be processed, calculating a second time required to transmit a subset of the second data that has not been uploaded to the third party to the third party, and sending the second time to the first participant;

[0031] receiving the task type of the to-be-processed task returned by the first participant, and sending the subset of the second data that has not been uploaded to the third party to the third party when the to-be-processed task is of a centralized type, so that the third party can perform centralized computing on the basis of the second data and the first data uploaded to the third party by the first participant based on the task type to obtain a processing result; or performing decentralized computing in coordination with the third party and other participants through local task scheduling to obtain a processing result when the to-be-processed task is of a decentralized type;

[0032] The task type is determined by the first participant according to the first time required for the first participant to transmit the subset of the first data associated with the task to be processed that has not been uploaded to the third party to the third party, the second time of each other second participant, and a preset time threshold;

[0033] The task types include decentralized types and centralized types.

[0034] Optionally, the task scheduling method further includes:

[0035] When in an idle state, the local first data set is uploaded to the third party.

[0036] Optionally, when the task to be processed is of a distributed type, the step of obtaining the processing result by cooperating with a third party and other participants to perform distributed computing through local task scheduling includes:

[0037] If the decentralized type of the task to be processed is a horizontal federation type, sending first intermediate data to the third party; wherein the first intermediate data is generated by each participant based on a corresponding first model gradient, and the first model gradient is calculated by each participant based on a corresponding local model;

[0038] Receiving aggregated intermediate data sent by a third party, wherein the aggregated intermediate data is obtained by the third party aggregating the first intermediate data;

[0039] updating model parameters based on the aggregated intermediate data to obtain updated local models, and continuing to generate new first intermediate data based on the respective updated local models;

[0040] Iterate and return to the step of sending the first intermediate data to the third party until the first preset completion condition is met, and use the data result corresponding to the first preset completion condition as the processing result.

[0041] Optionally, when the task to be processed is of a distributed type, the step of obtaining the processing result by cooperating with a third party and other participants to perform distributed computing through local task scheduling includes:

[0042] If the decentralized type of the task to be processed is a vertical federation type, encrypting and masking the second intermediate data to obtain an intermediate result, and sending the intermediate result to the third party;

[0043] The second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model;

[0044] Receiving the decrypted intermediate result sent by the third party after decrypting the intermediate result, updating the respective model parameters based on the decrypted intermediate result, obtaining the respective updated local models, and continuing to generate new intermediate results based on the respective updated local models;

[0045] The process iterates back to the step of encrypting the second intermediate data and performing mask processing to obtain an intermediate result until a second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result.

[0046] The present application also provides a task scheduling method based on the time consumption of data transmission, which is applied to a third party. The task scheduling method includes:

[0047] When the task type of the task to be processed initiated by the first participant is a centralized type, a centralized operation is performed on the first data and the second data to obtain a processing result, wherein the first data is the data associated with the task to be processed uploaded by the first participant to the third party, and the second data is the data associated with the task to be processed uploaded by each second participant to the third party; or

[0048] When the task type of the task to be processed initiated by the first participant is a distributed type, the processing result is obtained by cooperating with other participants through local calculation to perform distributed calculation;

[0049] Among them, the task type of the pending task initiated by the first participant is determined by the first participant based on the first time required for the first participant to transmit a subset of the first data of the pending task that has not been uploaded to the third party to the third party, the second time required for each other second participant to transmit a subset of the second data of the pending task that has not been uploaded to the third party to the third party, and a preset time threshold.

[0050] Optionally, when the task type of the task to be processed initiated by the first participant is a centralized type, performing centralized calculation in combination with the first data and the second data to obtain a processing result includes:

[0051] When the task type of the task to be processed initiated by the first participant is a centralized type, the first data and the second data are combined, the centralized task is scheduled by adopting a centralized scheduling mode, and the first participant and each second participant perform privacy calculation on the task to be processed according to the target scheduling mode corresponding to the task type to obtain a processing result.

[0052] Optionally, the step of combining the first data and the second data, scheduling centralized tasks by adopting a centralized scheduling mode, and cooperating with the first participant and each second participant to perform privacy calculation on the to-be-processed task according to a target scheduling mode corresponding to the task type to obtain a first processing result includes:

[0053] Determine whether the complete set of first data of the first participant and the complete set of second data of each second participant are stored locally;

[0054] If not, scheduling the first participant to encrypt and upload the subset of the first data that has not been uploaded to the third party, and scheduling other second participants to encrypt and upload the subset of the second data that has not been uploaded to the third party, and storing the full set of the first data and the full set of the second data;

[0055] The entire set of the first data and the entire set of the second data are retrieved locally, and the tasks to be processed are centrally processed to obtain processing results.

[0056] Optionally, when the task type of the task to be processed initiated by the first participant is a distributed type, local calculation is performed to cooperate with other participants to perform distributed calculations to obtain processing results, including:

[0057] When the task type of the task to be processed initiated by the first participant is a decentralized type, the decentralized type tasks are scheduled by adopting a decentralized scheduling mode, and the first participant and the second participant perform privacy calculations on the task to be processed according to the target scheduling mode corresponding to the task type, and obtain processing results.

[0058] Optionally, the method of scheduling the decentralized type tasks by adopting a decentralized scheduling mode, and cooperating with the first participant and the second participant to perform privacy calculation on the to-be-processed tasks according to the target scheduling mode corresponding to the task type, and obtaining the processing result includes:

[0059] If the decentralized type of the task to be processed is a horizontal federated decentralized type, receiving first intermediate data encrypted and sent by each participant;

[0060] The first intermediate data is generated by each participant based on the corresponding first model gradient, and the first model gradient is obtained by each participant performing privacy calculation based on the corresponding local model;

[0061] aggregating the first intermediate data to obtain aggregated intermediate data;

[0062] Sending the aggregated intermediate data to each participant, so that each participant updates their own model parameters based on the aggregated intermediate data, obtains their own updated local models, and performs privacy calculations based on their own updated local models to continue to generate new first intermediate data;

[0063] Iterate and return to the step of receiving the first intermediate data sent by each participant until the first preset completion condition is met, and take the data result corresponding to the first preset completion condition as the processing result.

[0064] Optionally, the method of scheduling the decentralized type tasks by adopting a decentralized scheduling mode, and cooperating with the first participant and the second participant to perform privacy calculation on the to-be-processed tasks according to the target scheduling mode corresponding to the task type, and obtaining the processing result includes:

[0065] If the decentralized type of the task to be processed is a vertical federation type, receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data;

[0066] The second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model;

[0067] Decrypting the intermediate result, and sending the decrypted intermediate result to the corresponding participant, so that each participant can update their own model parameters based on the intermediate result, obtain their own updated local models, and continue to generate new intermediate results based on their own updated local models;

[0068] Iterate and return to the step of receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data, until the second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result.

[0069] The present application also provides an electronic device, which is a physical node device, and includes: a memory, a processor, and a program of the task scheduling method stored in the memory and executable on the processor. When the program of the task scheduling method is executed by the processor, the steps of the task scheduling method described above can be implemented.

[0070] The present application also provides a storage medium, on which is stored a program for implementing the above-mentioned task scheduling method based on the time consumption of data transmission. When the program of the task scheduling method is executed by a processor, the steps of the task scheduling method as described above are implemented.

[0071] The present application also provides a product, which is a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned task scheduling method based on time consumption of data transmission.

[0072] The hybrid architecture system and task scheduling method based on data transmission time provided by the present application are compared with the prior art in which users manually select specific calculation methods, which easily causes waste of privacy computing resources. In the present application, a task to be processed is submitted, the first data associated with the task to be processed is determined, and the first time required to transmit the subset of the first data that has not been uploaded to the third party to the third party is calculated; the second time required to transmit the subset of the second data associated with the task to be processed that has not been uploaded to the third party to the third party is received from each other second participant; the task type of the task to be processed is determined based on the first time, the second time and a preset time threshold, wherein the task type includes a decentralized type and a centralized type; when the task to be processed is of a centralized type, the subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party by the second participant after receiving the task type to perform centralized calculations to obtain processing results; or when the task to be processed is of a decentralized type, local task scheduling is used to cooperate with the third party and with other participants to perform decentralized calculations to obtain processing results. It can be understood that in this application, a specific calculation method is not selected manually, but through clear judgment of transmission time, the least time-consuming, that is, the optimal calculation method is selected to avoid wasting privacy computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0075] Figure 1 This is a flowchart of the first embodiment of the task scheduling method based on the time consumption of data transmission of the present application;

[0076] Figure 2 This is a detailed flowchart of step S30 in the first embodiment of the task scheduling method based on data transmission time consumption of this application;

[0077] Figure 3 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

[0078] Figure 4 This is a schematic diagram of a first scenario of performing centralized operations on centralized type tasks involved in the task scheduling method based on time-consuming data transmission in the present application;

[0079] Figure 5 A schematic diagram of a second scenario of performing distributed computing when the distributed type of the task to be processed is a horizontal federated distributed type involved in the task scheduling method based on the time consumption of data transmission of the present application;

[0080] Figure 6 A schematic diagram of a third scenario of performing distributed computing when the distributed type of the task to be processed is a vertical federated distributed type involved in the task scheduling method based on the time consumption of data transmission of the present application;

[0081] Figure 7 This is a schematic diagram of the first full process of task scheduling based on data transmission time consumption involved in the fifth embodiment of the task scheduling method based on data transmission time consumption of the present application;

[0082] Figure 8 This is a second full process diagram of task scheduling based on data transmission time consumption involved in the thirteenth embodiment of the task scheduling method based on data transmission time consumption of the present application;

[0083] Fig. 9 A schematic diagram of a hybrid architecture system of the present application.

[0084] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0085] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0086] Embodiment 1

[0087] The embodiment of the present application provides a task scheduling system based on a hybrid architecture system, wherein the hybrid architecture system includes:

[0088] A first participant, the first participant is used to: initiate a task to be processed, determine first data associated with the task to be processed, and calculate a first time required to transmit a subset of the first data that has not been uploaded to a third party to the third party; also used to receive a second time required to transmit a subset of the second data associated with the task to be processed that has not been uploaded to a third party from each other second participant, and determine a task type of the task to be processed based on the first time, the second time and a preset time threshold, wherein the task type includes a decentralized type and a centralized type; also used to send the subset of the first data that has not been uploaded to a third party to the third party when the task to be processed is of a centralized type, or to coordinate with the third party and other participants to perform decentralized computing through local task scheduling when the task to be processed is of a decentralized type;

[0089] The second participant is used to: after detecting that the first participant has initiated a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant; and also to send the subset of the second data that has not been uploaded to the third party to the third party when the task to be processed is of a centralized type, or to coordinate the third party and other participants to perform distributed computing through local task scheduling when the task to be processed is of a decentralized type;

[0090] A third party, the third party is used to: when the task to be processed is of a centralized type, combine the first data and the second data to perform centralized computing, or when the task to be processed is of a decentralized type, cooperate with other participants to perform decentralized computing through local computing.

[0091] In this embodiment, there may be multiple second participants.

[0092] In this embodiment, the first participant initiates a task to be processed, which is a privacy computing task or a model training task. Specifically, it can be an image modeling task based on federated learning, a recommendation (prediction) modeling task based on federated learning, a classification modeling task based on federated learning, a query modeling task based on federated learning, etc., which are not specifically limited here. After the first participant initiates the task to be processed, the first data associated with the task to be processed is determined, and the first time required to transmit the subset of the first data that has not been uploaded to the third party to the third party is calculated. In this embodiment, the first time is associated with the first effective bandwidth between the first participant and the third party, and is the first time required to transmit the subset that has not been uploaded to the third party to the third party. Specifically, if the first data associated with the task to be processed locally on the first participant has been uploaded, the upload time is 0. If not all of them are uploaded, the first participant counts the waiting_size of the unuploaded part of the first data (the size of the subset of the first data that has not been uploaded to the third party), and then determines the first time consumption based on the size of the unuploaded first data and the first effective bandwidth; the first participant is also used to receive the second time consumption (direct reception) required for the subset of the second data associated with the task to be processed that has not been uploaded to the third party to be transmitted to the third party, and determines the task type of the task to be processed based on the first time consumption, the second time consumption and the preset time consumption threshold, wherein the task type includes a decentralized type and a centralized type, and the method for determining the task type The method includes, for example: selecting the maximum time consumption from the first time consumption and the second time consumption as the first data transmission time consumption; if the first data transmission time consumption is greater than a preset time consumption threshold, determining that the task type of the task to be processed is a distributed type; if the first data transmission time consumption is less than or equal to the preset time consumption threshold, determining that the task type of the task to be processed is a centralized type; the first participant is also used to send a subset of the first data that has not been uploaded to the third party to the third party when the task to be processed is a centralized type, or to coordinate with the third party and other participants to perform distributed computing through local task scheduling when the task to be processed is a distributed type. In this embodiment, the first participant can also upload the local data set to the third party when it is idle.

[0093] In this embodiment, the second participant is used to: after detecting that the first participant has initiated a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant; it is also used to send the subset of the second data that has not been uploaded to the third party to the third party when the task to be processed is of a centralized type, or to coordinate the third party and other participants to perform distributed computing through local task scheduling when the task to be processed is of a distributed type. Among them, the second data is the data of the second participant's local participation in the privacy calculation of the task to be processed. In this embodiment, the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party is calculated, and the second time is associated with the second effective bandwidth between the third party and the second participant, and is the second time required to transmit the subset that has not been uploaded to the third party to the third party. Specifically, if all the local data of the second participant has been uploaded, the upload time is 0. If not all of them are uploaded, the second participant counts the waiting_size of the unuploaded portion of the second data (the size of the subset of the second data that has not been uploaded to the third party), and then determines the second time consumption based on the size of the unuploaded portion of the second data and the second effective bandwidth. In this embodiment, the second participant can also upload the local data set to the third party when it is in an idle state.

[0094] In this embodiment, the third party is used to: when the task to be processed is of a centralized type, combine the first data and the second data to perform centralized computing, or when the task to be processed is of a decentralized type, cooperate with other participants to perform decentralized computing through local computing.

[0095] In a specific implementation, the first participant includes a first scheduling module, a first computing module, a first communication module and a task type recommendation module, wherein the first communication module is used to communicate with the second participant and / or the third party; the task type recommendation module is used to initiate a task to be processed, determine the first data associated with the task to be processed, and calculate the first time required to transmit the subset of the first data that has not been uploaded to the third party to the third party, and to receive the second time from each other second participant, and determine the task type of the task to be processed based on the first time, the second time and a preset time threshold; the first scheduling module is used to schedule the first computing module to perform local calculations through local tasks when the task to be processed is a distributed type to cooperate with the third party and other participants to perform distributed calculations, or to send the subset of the first data that has not been uploaded to the third party to the third party through the first communication module when the task to be processed is a centralized type.

[0096] In this embodiment, if Fig. 9As shown, the first participant includes a first scheduling module, a first calculation module, a first communication module and a task type recommendation module. In addition, the first participant may also include a first storage module.

[0097] In this embodiment, the first communication module is used to communicate with the second participant and / or the third party.

[0098] The task type recommendation module is used to: initiate tasks to be processed (including modeling, prediction, query and other tasks), determine the first data associated with the tasks to be processed, and calculate the first time required to transmit the subset of the first data that has not been uploaded to the third party to the third party, and receive the second time of each other second participant, and determine the task type of the tasks to be processed based on the first time, the second time and the preset time threshold; when the tasks to be processed are of a centralized type, send the first data set to the third party through the first communication module.

[0099] The first computing module is used to send the first intermediate data to the third party if the decentralized type of the task to be processed is a horizontal federation type; wherein the first intermediate data is generated by each participant based on the corresponding first model gradient, and the first model gradient is calculated by each participant based on the corresponding local model; receive the aggregated intermediate data sent by the third party, wherein the aggregated intermediate data is obtained by aggregating the first intermediate data by the third party; update the model parameters based on the aggregated intermediate data to obtain the updated local model, and continue to generate new first intermediate data based on the respective updated local models; iterate and return to the step of sending the first intermediate data to the third party until the first preset completion condition is met, and use the data result corresponding to the time when the first preset completion condition is met as the processing result.

[0100] The first computing module is used for, if the decentralized type of the task to be processed is a vertical federation type, encrypting and masking the second intermediate data to obtain an intermediate result, and sending the intermediate result to the third party; wherein the second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model; receiving the decrypted intermediate result sent by the third party after decrypting the intermediate result, updating the respective model parameters based on the decrypted intermediate result, and obtaining the respective updated local models, and continuing to generate new intermediate results based on the respective updated local models; iteratively returning to the step of encrypting and masking the second intermediate data to obtain the intermediate result until the second preset completion condition is met, and taking the data result corresponding to the second preset completion condition as the processing result.

[0101] The first scheduling module is used to schedule the first computing module to perform local calculations through local tasks to cooperate with a third party and other participants to perform distributed calculations when the task to be processed is of a distributed type, or to send a subset of the first data that has not been uploaded to the third party to the third party through the first communication module when the task to be processed is of a centralized type.

[0102] The first storage module is used to store the task information of the task to be processed and the first data.

[0103] The second participant includes a second scheduling module, a second computing module and a second communication module. The second communication module is used to communicate with the first participant and / or the third party. The second scheduling module is used to: after detecting that the first participant initiates a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant through the second communication module, and when the task to be processed is a distributed type, schedule the second computing module through a local task to cooperate with the third party and other participants to perform distributed computing, or when the task to be processed is a centralized type, send the subset of the second data that has not been uploaded to the third party to the third party through the second communication module.

[0104] In this embodiment, if Fig. 9 As shown, the second participant includes a second scheduling module, a second calculation module and a second communication module. In addition, the second participant may also include a second storage module.

[0105] In this embodiment, the second scheduling module is used to determine the second data associated with the task to be processed after detecting that the first participant has initiated the task to be processed, and calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party; send the second time to the second communication module, which is used to schedule the second calculation module through local tasks to cooperate with the third party and other participants to perform distributed computing when the task to be processed is a distributed type, or send the subset of the second data that has not been uploaded to the third party to the second communication module when the task to be processed is a centralized type.

[0106] In this embodiment, the second communication module is used to send the second time consumption to the first participant, and is also used to send the subset of the second data that has not been uploaded to the third party to the third party when the task to be processed is of a centralized type.

[0107] In one embodiment, the second computing module is used to send the first intermediate data to the third party when the decentralized type of the task to be processed is a horizontal federation type; wherein the first intermediate data is generated by each participant based on the corresponding first model gradient, and the first model gradient is calculated by each participant based on the corresponding local model; receive the aggregated intermediate data sent by the third party, wherein the aggregated intermediate data is obtained by aggregating the first intermediate data by the third party; update the model parameters based on the aggregated intermediate data to obtain the updated local model, and continue to generate new first intermediate data based on the respective updated local models; iterate and return to the step of sending the first intermediate data to the third party until the first preset completion condition is met, and use the data result corresponding to the time when the first preset completion condition is met as the processing result.

[0108] In another embodiment, the second computing module is used to encrypt and mask the second intermediate data to obtain an intermediate result when the distributed type of the task to be processed is a vertical federation type, and send the intermediate result to the third party; wherein the second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model; receive the decrypted intermediate result sent by the third party after decrypting the intermediate result, update their respective model parameters based on the decrypted intermediate result, and obtain their respective updated local models, and continue to generate new intermediate results based on their respective updated local models; iterate and return to the step of encrypting and masking the second intermediate data to obtain the intermediate result until the second preset completion condition is met, and take the data result corresponding to the second preset completion condition as the processing result.

[0109] The second storage module is used to store second data.

[0110] Furthermore, if Fig. 9 As shown, the third party includes a third scheduling module, a third computing module and a third communication module, and the third communication module is used to communicate with the first participant and / or the second participant; the third scheduling module is used to: when the task to be processed is of a centralized type, schedule the third computing module to perform centralized calculations in combination with the first data and the second data, or when the task to be processed is of a distributed type, schedule the third computing module to perform local calculations through local tasks to cooperate with other participants to perform distributed calculations.

[0111] In a specific implementation, the third computing module is used to perform centralized computing processing on the task to be processed based on the full set of the first data and the full set of the second data to obtain a processing result when the decentralized type of the task to be processed is a centralized type; the third storage module is used to store the first data and the second data, as well as to store various data sets uploaded by each participant to a third party when they are idle.

[0112] The third computing module is also used to receive the first intermediate data encrypted and sent by each participant when the decentralized type of the task to be processed is a horizontal federated decentralized type; wherein the first intermediate data is generated by each participant based on the corresponding first model gradient, and the first model gradient is obtained by each participant performing privacy calculation based on the corresponding local model; the first intermediate data is aggregated to obtain aggregated intermediate data; the aggregated intermediate data is sent to each participant so that each participant can update their own model parameters based on the aggregated intermediate data, obtain their own updated local models, and perform privacy calculation based on their own updated local models to continue to generate new first intermediate data; iteratively return to the step of receiving the first intermediate data sent by each participant until the first preset completion condition is met, and the data result corresponding to the first preset completion condition is used as the processing result. Wherein, the first preset completion condition can be that the corresponding first preset loss function to be processed converges or the corresponding first training times reach the first preset times; the data result corresponding to the first preset completion condition is the model obtained after training.

[0113] The third computing module is also used to receive the intermediate results sent by each participant after encrypting and masking the second intermediate data when the decentralized type of the task to be processed is a vertical federation type; wherein the second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model; decrypt the intermediate result, and send the decrypted intermediate result to the corresponding participant, so that each participant can update their own model parameters based on the intermediate result, and obtain their own updated local models, and continue to generate new intermediate results based on their own updated local models; iterate and return to the step of receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data, until the second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result. The second preset completion condition is that the number of training times reaches the second preset number of times, or the second preset loss function converges; the processing result can be a model obtained after vertical federation, which can be used to predict the accuracy.

[0114] Embodiment 2

[0115] The present application embodiment provides a task scheduling method based on the time consumption of data transmission, referring to Figure 1 The task scheduling method based on the time consumption of data transmission provided in this embodiment is based on the above hybrid architecture system and is applied to any first participant in the participant subsystem who initiates a task to be processed. The participant subsystem also includes at least one second participant associated with the task to be processed. The task scheduling method includes:

[0116] Step S10, submitting the task to be processed, determining first data associated with the task to be processed, and calculating a first time required to transmit a subset of the first data that has not been uploaded to the third party to the third party;

[0117] Step S20, receiving a second time required for transmitting a subset of the second data associated with the task to be processed that has not been uploaded to the third party from each other second participant to the third party;

[0118] Step S30, determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold, wherein the task type includes a decentralized type and a centralized type;

[0119] Step S40, when the task to be processed is of a centralized type, the subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party after the second participant receives the task type to perform centralized calculations to obtain processing results; or when the task to be processed is of a distributed type, local task scheduling is used to cooperate with the third party and other participants to perform distributed calculations to obtain processing results.

[0120] In this embodiment, the first participant adds a judgment of the transmission time corresponding to the pending task based on federated learning, and then determines the task type of the pending task, and then selects the target scheduling mode of the pending task from the preset different scheduling modes (at least including a distributed scheduling mode and a centralized scheduling mode, and the corresponding task types include a distributed type and a centralized type) to achieve task processing of the pending task. It can be understood that the specific calculation method is not selected by humans, but through a clear judgment of the transmission time, the optimal calculation method with the least time consumption (least waste of time resources) is selected to avoid wasting privacy computing resources.

[0121] The specific steps are as follows:

[0122] Step S10, submitting the task to be processed, determining first data associated with the task to be processed, and calculating a first time required to transmit a subset of the first data that has not been uploaded to the third party to the third party;

[0123] In this embodiment, it should be noted that the task scheduling method based on data transmission time consumption can be applied to a task scheduling device based on a hybrid architecture system. The task scheduling device based on the hybrid architecture system belongs to an electronic device, and the electronic device is subordinate to a task scheduling system based on a hybrid architecture system.

[0124] In recent years, the integrated application of data has driven all walks of life towards digitalization, networking and intelligence. In the process of all walks of life moving towards digitalization, networking and intelligence, issues such as data security and personal privacy protection have increasingly received widespread attention from the society. That is, how to achieve data integration under the premise of compliance (ensuring data security and ensuring personal privacy protection) has become a difficult problem that needs to be solved urgently.

[0125] Privacy computing refers to a technology and system for joint computing by two or more parties. The parties collaborate to conduct joint machine learning and joint analysis on their data without disclosing their own data. The participants in privacy computing can be different departments of the same organization or different organizations. Under the privacy computing framework, the data of the participants will not be exported in plain text. While protecting data security, multi-source data cross-domain cooperation is achieved to solve the problem of data protection and integrated application.

[0126] At present, privacy computing mainly includes federated learning computing. Federated learning refers to a method of performing machine learning by uniting different participants. A round of parameter update in federated learning is divided into two steps: (a) each participant only uses its own data to train the machine learning model and sends model parameter updates to a central coordinator; (b) the coordinator integrates the model updates received from different participants (for example, taking the average) and redistributes the integrated model parameter updates to each participant. In federated learning, participants do not need to expose their data to other participants or coordinators. Therefore, federated learning can well protect user privacy and ensure data security.

[0127] Federated learning includes vertical federated learning computing and horizontal federated learning computing. Vertical federated learning is sample-aligned federated learning (Sample-Aligned Federated Learning), which is suitable for situations where the participants' training sample IDs overlap a lot but the data features overlap less. It requires multiple parties to collaborate to complete model training and optimization under a secure and confidential framework.

[0128] Horizontal Federated Learning is suitable for situations where the data features of participants' training samples overlap a lot but there are fewer user IDs. It requires multiple parties to collaborate to complete model training and optimization under a secure and confidential framework.

[0129] In this embodiment, the specific application scenario may be:

[0130] During the privacy computing process, different computing methods can be selected. Different computing methods have different data transmission and data storage capacities. In the prior art, users manually select specific computing methods. However, the manually selected computing method is often not the most optimal, resulting in a waste of privacy computing resources.

[0131] In this embodiment, a judgment of the transmission time corresponding to the pending task based on federated learning is added, and then the target scheduling mode of the pending task is selected from different preset scheduling modes (including at least a distributed scheduling mode and a centralized scheduling mode) to realize task processing of the pending task. It can be understood that the specific calculation method is not selected by humans based on their own assumptions, but through a clear judgment of the transmission time, the calculation method with the least time consumption, that is, the optimal one, is selected to avoid wasting privacy computing resources.

[0132] In this embodiment, the first participant submits a task to be processed, determines first data associated with the task to be processed, and calculates a first time required to transmit a subset of the first data that has not been uploaded to the third party to the third party.

[0133] In this embodiment, the task to be processed is a privacy computing task, or a model training task. Specifically, it can be an image modeling task based on federated learning, a recommendation (prediction) modeling task based on federated learning, a classification modeling task based on federated learning, a query modeling task based on federated learning, etc., which are not specifically limited here.

[0134] In this embodiment, it is a to-be-processed task submitted by a first participant, and the first participant can communicate with a task scheduling system based on a hybrid architecture system, or the first participant is a part of a task scheduling system based on a hybrid architecture system, and the task scheduling system based on a hybrid architecture system also includes multiple other second participants and a coordinator. In this embodiment, the first participant is taken as a part of a task scheduling system based on a hybrid architecture system for specific description.

[0135] In this embodiment, the task scheduling system based on the hybrid architecture system may also include a third party.

[0136] In this embodiment, the distributed scheduling mode is: the original data is located in the local server of each participant. After the participants complete the calculation locally, they exchange the ciphertext data of the intermediate results through the network to complete the calculation of the task to be processed. Figure 5 as well as Figure 6 shown.

[0137] In this embodiment, the data centralized mode is: each participant transmits the ciphertext of the original data to a third party (centralized computing environment), and the third party (centralized computing environment) returns the calculation result to the task initiator, i.e., the first participant, after completing the calculation locally. Figure 4 shown.

[0138] In this embodiment, the task scheduling system based on the hybrid architecture system can change the target scheduling mode of the task to be processed, thereby completing the processing of the task to be processed.

[0139] Specifically, in this embodiment, the task scheduling system based on the hybrid architecture system includes a decentralized and centralized hybrid architecture, thereby achieving a change in the target scheduling mode of the task to be processed.

[0140] In this embodiment, the first participant initiates a task to be processed, determines first data associated with the task to be processed, and calculates a first time required to transmit a subset of the first data that has not been uploaded to a third party to the third party.

[0141] In this embodiment, the first data is data of the first participant participating locally in the privacy computing of the task to be processed.

[0142] In this embodiment, a first time consumption required to transmit the subset of the first data that has not been uploaded to the third party to the third party is calculated.

[0143] In this embodiment, the first time consumption is associated with a first effective bandwidth between the third party and the first participant, and is a first time consumption required for the subset of the first data that has not been uploaded to the third party to be transmitted to the third party.

[0144] Specifically, a network command (such as iperf) of a task scheduling system based on the hybrid architecture system calculates or determines a first effective bandwidth from a third party to a first participant.

[0145] Specifically, if all local data sets of the first participant have been uploaded, the upload time is 0. If not all have been uploaded, the first participant counts the waiting_size (the size of the first data not uploaded) of the unuploaded portion of the first data, and then determines the first time based on the size of the unuploaded portion of the first data and the first effective bandwidth.

[0146] In this embodiment, the first participant may also actively upload the local data set to the third party when in an idle state.

[0147] Step S20, receiving a second time required for transmitting a subset of the second data associated with the task to be processed that has not been uploaded to the third party from each other second participant to the third party;

[0148] In this embodiment, the second data is data of the second participant participating locally in the privacy calculation of the task to be processed.

[0149] In this embodiment, the first participant, as the initiator, initiates the task to be processed, and at this time, the other second participants detect the task to be processed.

[0150] In this embodiment, a second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party is calculated.

[0151] In this embodiment, the second time consumption is associated with a second effective bandwidth between the third party and the second party, and is a second time consumption required for the subset that has not been uploaded to the third party to be transmitted to the third party.

[0152] Specifically, a network command (such as iperf) of a task scheduling system based on the hybrid architecture system calculates or determines a second effective bandwidth from the third party to the second participant.

[0153] Specifically, if the local data of the second participant has been uploaded, the uploading time is 0.

[0154] If not all of them are uploaded, the second participant counts the waiting_size of the unuploaded portion of the second data (the size of the subset of the second data that has not been uploaded to the third party), and then determines the second time consumption based on the size of the unuploaded portion of the second data and the second effective bandwidth.

[0155] In this embodiment, the second participant may actively upload the local data set to the third party when in an idle state.

[0156] Step S30, determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold, wherein the task type includes a decentralized type and a centralized type;

[0157] In this embodiment, the task type of the task to be processed is determined based on the first time consumption, the second time consumption and the preset time consumption threshold. Specifically, the larger time consumption is selected from the first time consumption and the second time consumption, and the larger time consumption is compared with the preset time consumption threshold to obtain a comparison result. Based on the comparison result, the task type of the task to be processed is determined.

[0158] Reference Figure 2 , wherein the step of determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold comprises:

[0159] Step S31, selecting a maximum time consumption from the first time consumption and the second time consumption as a first data transmission time consumption;

[0160] Step S32, if the first data transmission time is greater than a preset time threshold, determining that the task type of the task to be processed is a distributed type;

[0161] Step S33: If the first data transmission time is less than or equal to a preset time threshold, it is determined that the task type of the task to be processed is a centralized type.

[0162] In this embodiment, the overall flow chart is as follows: Figure 7 shown.

[0163] In this embodiment, since each participant communicates synchronously with the third party, the maximum time consumption is selected from the first time consumption and the second time consumption as the first data transmission time consumption associated with the task to be processed.

[0164] Step S40, when the task to be processed is of a centralized type, the subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party after the second participant receives the task type to perform centralized calculations to obtain processing results; or when the task to be processed is of a distributed type, local task scheduling is used to cooperate with the third party and other participants to perform distributed calculations to obtain processing results.

[0165] In this embodiment, the scheduling model of the distributed type of tasks to be processed is a distributed scheduling mode.

[0166] In this embodiment, the scheduling model of the centralized type of tasks to be processed is a centralized scheduling mode.

[0167] That is, in this embodiment, the task scheduling system or platform based on the hybrid architecture system is mainly divided into two scheduling modes:

[0168] Distributed scheduling mode: The original data is located on each participant's own server. After the participant completes the calculation locally, the encrypted data of the intermediate results are exchanged with each other through the network.

[0169] Centralized scheduling mode: Each participant transmits the ciphertext of the original data to a third party (centralized computing environment). After the third party (centralized computing environment) completes the calculation locally, it returns the calculation result to the task initiator, i.e. the first participant.

[0170] Of course, different scheduling modes may also include other scheduling modes, which are not specifically limited here.

[0171] In this embodiment, if the first data transmission time is greater than a preset time threshold, a distributed scheduling mode is selected from preset different scheduling modes as the target scheduling mode of the task to be processed. At this time, the first participant performs distributed computing by coordinating with the third party and other participants through local task scheduling to obtain a processing result. That is, when the task to be processed is of a distributed type, the first participant performs distributed computing by coordinating with the third party and other participants through local task scheduling to obtain a processing result.

[0172] That is, it is obvious that since the first data transmission time is greater than the preset time threshold, the centralized scheduling mode takes too much time and causes excessive waste of resources. Therefore, the decentralized scheduling mode is selected as the target scheduling mode for the task to be processed.

[0173] If the first data transmission time is less than or equal to the preset time threshold, a centralized scheduling mode is selected from the preset different scheduling modes as the target scheduling mode of the task to be processed. The subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party after the second participant receives the task type (part of the second data may be uploaded to the third party by the second participant in the idle state) to perform centralized calculations to obtain processing results. That is, when the task to be processed is of a centralized type, the subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party after the second participant receives the task type (part of the second data may be uploaded to the third party by the second participant in the idle state) to perform centralized calculations to obtain processing results.

[0174] If the first data transmission time is less than or equal to the preset time threshold, a centralized scheduling mode is selected from different preset scheduling modes as the target scheduling mode of the task to be processed, that is,

[0175] Obviously, since the first data transmission time is less than or equal to the preset time threshold, at this time, the centralized scheduling mode consumes less time and will not cause excessive waste of resources. Therefore, the centralized scheduling mode is selected as the target scheduling mode for the task to be processed.

[0176] The present application provides a hybrid architecture system and a task scheduling method based on data transmission time consumption. Compared with the prior art in which users manually select a specific calculation method, which easily causes a waste of privacy computing resources, in the present application, a task to be processed is submitted, the first data associated with the task to be processed is determined, and the first time consumption required to transmit the subset of the first data that has not been uploaded to the third party to the third party is calculated; the second time consumption required to transmit the subset of the second data associated with the task to be processed that has not been uploaded to the third party to the third party is received from each other second participant; the task type of the task to be processed is determined based on the first time consumption, the second time consumption and a preset time consumption threshold, wherein the task type includes a decentralized type and a centralized type; when the task to be processed is of a centralized type, the subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party by the second participant after receiving the task type to perform centralized calculations to obtain processing results; or when the task to be processed is of a decentralized type, local task scheduling is used to cooperate with the third party and with other participants to perform decentralized calculations to obtain processing results. It can be understood that in this application, a specific calculation method is not selected manually, but through clear judgment of transmission time, the least time-consuming, that is, the optimal calculation method is selected to avoid wasting privacy computing resources.

[0177] Embodiment 3

[0178] Further, based on the first embodiment of the present application, another embodiment of the present application is provided. In this embodiment, referring to Figure 8 The step of determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold comprises:

[0179] Step A1, obtaining the task information of the task to be processed;

[0180] In this embodiment, a specific method for determining the type of task to be processed is provided.

[0181] In this embodiment, the task information may be attribute information such as task name and task type.

[0182] Step A2, based on the task information, searching from preset historical tasks for a first historical task that matches the similarity of the task to be processed and is of a decentralized type;

[0183] In this embodiment, matching the similarity of tasks to be processed may refer to:

[0184] First, the similarity of the tasks to be processed is greater than a first preset similarity value;

[0185] Second, the similarity of the task information of the task to be processed is greater than a second preset similarity value.

[0186] Step A3, obtaining the intermediate transmission volume during the process of transmitting intermediate data between the first participant and other second participants in the process of processing the first historical task, and calculating the second data transmission time consumption based on the intermediate transmission volume and the current network status information, and using the second data transmission time consumption as the preset time consumption threshold;

[0187] Step A4, selecting a maximum time consumption from the first time consumption and the second time consumption as a first data transmission time consumption;

[0188] Step A5: if the first data transmission time is greater than or equal to the preset time threshold, determining that the type of the task to be processed is a distributed type;

[0189] Step A6: If the first data transmission time is less than the preset time threshold, it is determined that the type of the task to be processed is a centralized type.

[0190] In this embodiment, the intermediate transmission volume during the transmission of intermediate data by the first participant and other second participants in the process of processing the first historical task is obtained, and the second data transmission consumption is calculated based on the intermediate transmission volume and the current network status information, and the maximum time consumption is selected from the first time consumption and the second time consumption as the first data transmission consumption.

[0191] If the first data transmission time is greater than or equal to the second data transmission time, it is obvious that the type of the task to be processed is determined to be a distributed type, that is, compared with the distributed scheduling mode, the centralized scheduling mode is too time-consuming and causes too much waste of resources. Therefore, the distributed scheduling mode is selected as the target scheduling mode of the task to be processed. That is, at this time, the third party coordinates the first participant and other second participants to perform distributed computing through local task scheduling to obtain the processing result. That is, when the task to be processed is of a distributed type, the first participant and other second participants are coordinated to perform distributed computing through local task scheduling to obtain the processing result.

[0192] If the first data transmission time is less than the second data transmission time, it is obvious that the type of the task to be processed is determined to be a centralized type. That is, it is obvious that the centralized scheduling mode takes less time and will not cause excessive waste of resources. Therefore, the centralized scheduling mode is selected as the target scheduling mode of the task to be processed. That is, at this time, the third party performs centralized calculations through local task scheduling to obtain the processing results. That is, when the task to be processed is of a centralized type, centralized calculations are performed through local task scheduling to obtain the processing results.

[0193] In this embodiment, by obtaining the task information of the task to be processed; based on the task information, querying from the preset historical tasks a first historical task that matches the similarity of the task to be processed and whose task type is a distributed type; obtaining the intermediate transmission volume during the transmission of intermediate data by the first participant and other second participants in the processing of the first historical task, and calculating the second data transmission time based on the intermediate transmission volume and the current network status information; selecting the maximum time from the first time and the second time as the first data transmission time; if the first data transmission time is greater than or equal to the second data transmission time, determining that the type of the task to be processed is a distributed type; if the first data transmission time is less than the second data transmission time, determining that the type of the task to be processed is a centralized type. In this embodiment, accurate selection of a less time-consuming mode for task scheduling and processing is achieved, improving efficiency and reducing resource consumption.

[0194] Embodiment 4

[0195] Furthermore, the present application provides a task scheduling method based on data transmission time consumption, which is applied to any second participant in a participant subsystem that is passively associated with a task to be processed, wherein the participant subsystem also includes a first participant that initiates the task to be processed, and the task scheduling method includes:

[0196] Step A1, after detecting that the first participant initiates a task to be processed, determining second data associated with the task to be processed, calculating a second time required to transmit a subset of the second data that has not been uploaded to the third party to the third party, and sending the second time to the first participant;

[0197] Step A2, receiving the task type of the task to be processed returned by the first participant, and sending the subset of the second data that has not been uploaded to the third party to the third party when the task to be processed is of a centralized type, so that the third party can combine the second data and the first data uploaded to the third party by the first participant based on the task type to perform centralized computing to obtain a processing result; or when the task to be processed is of a decentralized type, performing decentralized computing in cooperation with the third party and other participants through local task scheduling to obtain a processing result;

[0198] The task type is determined by the first participant according to the first time required for the first participant to transmit the subset of the first data associated with the task to be processed that has not been uploaded to the third party to the third party, the second time of each other second participant, and a preset time threshold;

[0199] The task types include decentralized types and centralized types.

[0200] In this embodiment, the execution subject is any second participant in the participant subsystem, and there can be multiple second participants, and the second participant can also actively upload the local data set to the third party when it is idle. The specific implementation method of the first participant performing task scheduling is basically the same as the various embodiments of the task scheduling method described above, and will not be repeated here.

[0201] Embodiment 5

[0202] Further, based on the above-mentioned Embodiment 2, Embodiment 3 and Embodiment 4, another embodiment of the present application is provided. In this embodiment, when the task to be processed is of a distributed type, the steps of obtaining the processing result by cooperating with a third party and other participants to perform distributed computing through local task scheduling include:

[0203] Step B1: if the decentralized type of the task to be processed is a horizontal federation type, sending first intermediate data to the third party; wherein the first intermediate data is generated by each participant based on a corresponding first model gradient, and the first model gradient is calculated by each participant based on a corresponding local model;

[0204] Step B2, receiving aggregated intermediate data sent by a third party, wherein the aggregated intermediate data is obtained by the third party aggregating the first intermediate data;

[0205] Step B3, updating model parameters based on the aggregated intermediate data to obtain an updated local model, and continuing to generate new first intermediate data based on the respective updated local models;

[0206] Step B4, iteratively returns to the step of sending the first intermediate data to the third party until the first preset completion condition is met, and the data result corresponding to the first preset completion condition is taken as the processing result.

[0207] In this embodiment, specifically, Figure 5 As shown in the figure, the decentralized type of the task to be processed is the horizontal federation type, and the process of scheduling or executing the task is as follows:

[0208] (1) Participant 1 initiates the task as the initiator. Each participant calculates the model gradient (first model gradient) locally and uses encryption technologies such as homomorphic encryption, differential privacy, or secret sharing to mask the gradient information. The masked results are sent to a third party (aggregation server or coordinator).

[0209] (2) The third party (aggregation server or coordinator) performs a secure aggregation operation, such as using weighted averaging based on homomorphic encryption, to obtain an aggregated result (aggregated intermediate data), wherein the aggregated intermediate data is obtained by the third party aggregating the first intermediate data;

[0210] (3) The third party (aggregation server or coordinator) sends the aggregated results (aggregated intermediate data) to each participant;

[0211] (4) Each participant decrypts the received aggregated intermediate data (gradient result) and uses the decrypted gradient result to update their own model parameters to continue to generate the first intermediate data.

[0212] Iterate steps (1)-(4) until a first preset completion condition is met, and use the data result corresponding to the first preset completion condition as the processing result.

[0213] Among them, the first preset completion condition may be that the corresponding first preset loss function to be processed converges or the corresponding first training times reaches the first preset times.

[0214] In this embodiment, the data result corresponding to the first preset completion condition is the model obtained after training.

[0215] In this embodiment, after the task to be processed is processed (the privacy calculation reaches the first preset completion condition), a processing result is obtained, wherein the processing result can be a model obtained after horizontal federation training, which can be used to predict accuracy.

[0216] In this embodiment, the most effective computing method is used to calculate the distributed tasks of the horizontal federation type to avoid wasting privacy computing resources.

[0217] Embodiment 6

[0218] Further, based on the above-mentioned Embodiment 2, Embodiment 3, Embodiment 4 and Embodiment 5, another embodiment of the present application is provided. In this embodiment, when the task to be processed is of a distributed type, the steps of obtaining the processing result by cooperating with a third party and other participants to perform distributed computing through local task scheduling include:

[0219] Step C1: if the decentralized type of the task to be processed is a vertical federation type, encrypting and masking the second intermediate data to obtain an intermediate result, and sending the intermediate result to the third party;

[0220] The second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model;

[0221] Step C2, receiving the decrypted intermediate result sent by the third party after decrypting the intermediate result, updating the respective model parameters based on the decrypted intermediate result, and obtaining the respective updated local models, and continuing to generate new intermediate results based on the respective updated local models;

[0222] Step C3, iteratively returns to the step of encrypting the second intermediate data and obtaining the intermediate result after masking, until the second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result.

[0223] Specifically, Figure 6 As shown in the figure, the decentralized type of the task to be processed is the vertical federation type, and the process of scheduling or executing the task is as follows:

[0224] (1) Participant 1 initiates the task as the initiator. The coordinator creates a key pair and sends the public key to Participant 1 and Participant 2.

[0225] (2) Participant 1 and Participant 2 encrypt and exchange the intermediate processing results, which are used to help calculate the gradient and loss value;

[0226] (3) Party 1 and Party 2 calculate the encrypted gradient and add the additional mask respectively to obtain the intermediate result (Party 2 also calculates the encryption loss). Party 1 and Party 2 send the encrypted result to the third party (coordinator);

[0227] (4) The third party (coordinator) decrypts the gradient and loss information and sends the results to Party 1 and Party 2. Party 1 and Party 2 remove the mask on the gradient information and update the model parameters based on the gradient information.

[0228] In this embodiment, the second preset completion condition is that the number of training times reaches the second preset number of times, or the second preset loss function converges.

[0229] In this embodiment, after the task to be processed is processed (the privacy calculation reaches the second preset completion condition), a processing result is obtained, wherein the processing result can be a model obtained after vertical federation, which can be used to predict accuracy.

[0230] It can be understood that in this embodiment, the most effective computing method is used to calculate the distributed tasks of the vertical federation type to avoid wasting privacy computing resources.

[0231] Embodiment 7

[0232] Furthermore, the present application provides a task scheduling method based on data transmission time consumption, which is applied to a third party. The task scheduling method includes:

[0233] Step A1, when the task type of the task to be processed initiated by the first participant is a centralized type, performing centralized calculations on the first data and the second data to obtain a processing result, wherein the first data is the data associated with the task to be processed uploaded by the first participant to the third party, and the second data is the data associated with the task to be processed uploaded by each second participant to the third party; or

[0234] Step A2, when the task type of the task to be processed initiated by the first participant is a distributed type, local calculation is performed to cooperate with other participants to perform distributed calculations to obtain a processing result;

[0235] Among them, the task type of the pending task initiated by the first participant is determined by the first participant based on the first time required for the first participant to transmit a subset of the first data of the pending task that has not been uploaded to the third party to the third party, the second time required for each other second participant to transmit a subset of the second data of the pending task that has not been uploaded to the third party to the third party, and a preset time threshold.

[0236] In this embodiment, the method of scheduling the decentralized type tasks by adopting a decentralized scheduling mode, cooperating with the first participant and the second participant to perform privacy calculation on the to-be-processed tasks according to the target scheduling mode corresponding to the task type, and obtaining the processing result includes:

[0237] If the decentralized type of the task to be processed is a horizontal federated decentralized type, receiving first intermediate data encrypted and sent by each participant;

[0238] The first intermediate data is generated by each participant based on the corresponding first model gradient, and the first model gradient is obtained by each participant performing privacy calculation based on the corresponding local model;

[0239] aggregating the first intermediate data to obtain aggregated intermediate data;

[0240] Sending the aggregated intermediate data to each participant, so that each participant updates their own model parameters based on the aggregated intermediate data, obtains their own updated local models, and performs privacy calculations based on their own updated local models to continue to generate new first intermediate data;

[0241] Iterate and return to the step of receiving the first intermediate data sent by each participant until the first preset completion condition is met, and take the data result corresponding to the first preset completion condition as the processing result.

[0242] Among them, the first preset completion condition may be that the corresponding first preset loss function to be processed converges or the corresponding first training times reaches the first preset times.

[0243] In this embodiment, the data result corresponding to the first preset completion condition is the model obtained after training.

[0244] In this embodiment, after the task to be processed is processed (the privacy calculation reaches the first preset completion condition), a processing result is obtained, wherein the processing result can be a model obtained after horizontal federation training, which can be used to predict accuracy.

[0245] In this embodiment, the executor is a third party, and the specific implementation method of the third party performing distributed type (horizontally federated distributed type) task scheduling is basically the same as the various embodiments of distributed type (horizontally federated distributed type) task scheduling in the above-mentioned task scheduling method, and will not be repeated here.

[0246] In this embodiment, the most effective computing method is used to calculate the distributed tasks of the horizontal federation type to avoid wasting privacy computing resources.

[0247] In this embodiment, the method of scheduling the decentralized type tasks by adopting a decentralized scheduling mode, cooperating with the first participant and the second participant to perform privacy calculation on the to-be-processed tasks according to the target scheduling mode corresponding to the task type, and obtaining the processing result includes:

[0248] If the decentralized type of the task to be processed is a vertical federation type, receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data;

[0249] The second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model;

[0250] Decrypting the intermediate result, and sending the decrypted intermediate result to the corresponding participant, so that each participant can update their own model parameters based on the intermediate result, obtain their own updated local models, and continue to generate new intermediate results based on their own updated local models;

[0251] Iterate and return to the step of receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data, until the second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result.

[0252] In this embodiment, the second preset completion condition is that the number of training times reaches the second preset number of times, or the second preset loss function converges.

[0253] In this embodiment, after the task to be processed is processed (the privacy calculation reaches the second preset completion condition), a processing result is obtained, wherein the processing result can be a model obtained after vertical federation, which can be used to predict accuracy.

[0254] In this embodiment, the executor is a third party, and the specific implementation method of the third party performing distributed type (vertical federated distributed type) task scheduling is basically the same as the various embodiments of distributed type (vertical federated distributed type) task scheduling in the above-mentioned task scheduling method, and will not be repeated here.

[0255] In this embodiment, the most effective computing method is used to calculate the distributed tasks of the vertical federation type to avoid wasting privacy computing resources.

[0256] In this embodiment, when the task type of the task to be processed initiated by the first participant is a centralized type, a centralized operation is performed in combination with the first data and the second data to obtain a processing result, including:

[0257] When the task type of the task to be processed initiated by the first participant is a centralized type, the first data and the second data are combined, the centralized task is scheduled by adopting a centralized scheduling mode, and the first participant and each second participant perform privacy calculation on the task to be processed according to the target scheduling mode corresponding to the task type to obtain a processing result.

[0258] In this embodiment, the step of combining the first data and the second data, scheduling the centralized task by adopting the centralized scheduling mode, and cooperating with the first participant and each second participant to perform privacy calculation on the task to be processed according to the target scheduling mode corresponding to the task type, to obtain the processing result includes:

[0259] Step D1, determining whether the complete set of first data of the first participant and the complete set of second data of each second participant are stored locally;

[0260] Step D2: if not, schedule the first participant to encrypt and upload the subset of the first data that has not been uploaded to the third party, and schedule other second participants to encrypt and upload the subset of the second data that has not been uploaded to the third party, and store the full set of the first data and the full set of the second data;

[0261] Step D3, retrieve the complete set of the first data and the complete set of the second data locally, perform centralized computing on the task to be processed, and obtain a processing result.

[0262] In this embodiment, a centralized scheduling mode is adopted to schedule centralized type tasks, and a specific description is given by taking two participants as an example. The specific process may be:

[0263] The third party determines whether the complete set of the first data of the first participant and the complete set of the second data of each second participant are stored locally. If not, the first participant encrypts and uploads the subset of the first data that has not been uploaded to the third party to the third party, and the second participant encrypts and uploads the subset of the second data that has not been uploaded to the third party to the third party.

[0264] Or schedule the first participant to encrypt and upload the subset of the first data that has not been uploaded to the third party, and schedule other second participants to encrypt and upload the subset of the second data that has not been uploaded to the third party, and store the full set of the first data and the full set of the second data, that is, the third party loads the data of both parties from different participants and saves it (this can be done when the bandwidth is idle).

[0265] During the task execution process, the third party retrieves the full set of the first data and the full set of the second data from the local, performs centralized computing on the task to be processed, and obtains the processing result. That is, the third party (centralized computing environment) performs calculations completely based on the local data of the third party (centralized computing environment), which saves the transmission of intermediate data and has relatively low bandwidth requirements.

[0266] In this embodiment, it is determined whether the complete set of the first data of the first participant and the complete set of the second data of each second participant are stored locally; if not, the first participant is scheduled to encrypt and upload the subset of the first data that has not been uploaded to the third party, and the other second participants are scheduled to encrypt and upload the subset of the second data that has not been uploaded to the third party, and the complete set of the first data and the complete set of the second data are stored; the complete set of the first data and the complete set of the second data are retrieved locally, and the to-be-processed tasks are centrally processed to obtain the processing results. In this embodiment, the third party performs the computational processing in a centralized mode on the to-be-processed tasks.

[0267] In this embodiment, the execution subject is a third party, and the specific implementation method of the centralized type task scheduling performed by the third party is basically the same as the various embodiments of the centralized type task scheduling in the task scheduling method described above, and will not be repeated here.

[0268] Embodiment 8

[0269] An electronic device is provided, which is a physical node device, and includes: a memory, a processor, and a program stored in the memory for implementing the task scheduling method based on the time consumption of data transmission, wherein the memory is used to store the program for implementing the task scheduling method based on the time consumption of data transmission; the processor is used to execute the program for implementing the task scheduling method based on the time consumption of data transmission, so as to implement the steps of the task scheduling method in the above-mentioned embodiment.

[0270] Reference Figure 3 , Figure 3 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0271] like Figure 3 As shown, the electronic device may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0272] Optionally, the electronic device may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0273] Those skilled in the art will understand that Figure 3 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0274] like Figure 3 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, and a task scheduler based on a hybrid architecture system. The operating system is a program that manages and controls the hardware and software resources of an electronic device, and supports the operation of a task scheduler based on a hybrid architecture system and other software and / or programs. The network communication module is used to realize communication between components within the memory 1005, and communication with other hardware and software in a task scheduling system based on a hybrid architecture system.

[0275] exist Figure 3 In the electronic device shown, the processor 1001 is used to execute the task scheduling program based on the hybrid architecture system stored in the memory 1005 to implement the steps of any of the above-mentioned task scheduling methods based on data transmission time consumption.

[0276] The specific implementation of the electronic device of the present application is basically the same as the above-mentioned embodiments of the task scheduling method based on the time consumption of data transmission, and will not be repeated here.

[0277] Embodiment 9

[0278] An embodiment of the present application provides a storage medium, and the storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of the above-mentioned task scheduling method based on data transmission time consumption.

[0279] The specific implementation of the storage medium of the present application is basically the same as the above-mentioned embodiments of the task scheduling method based on the time consumption of data transmission, and will not be repeated here.

[0280] Embodiment 10

[0281] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned task scheduling method based on data transmission time consumption.

[0282] The specific implementation of the computer program product of the present application is basically the same as the various embodiments of the task scheduling method based on data transmission time consumption described above, and will not be repeated here.

[0283] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0284] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0285] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0286] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A hybrid architecture system, characterized in that: The hybrid architecture system includes: A first participant, the first participant is used to: initiate a task to be processed, determine first data associated with the task to be processed, and calculate a first time required to transmit a subset of the first data that has not been uploaded to a third party to the third party; also used to receive a second time required to transmit a subset of the second data associated with the task to be processed that has not been uploaded to a third party from each other second participant, and determine a task type of the task to be processed based on the first time, the second time and a preset time threshold, wherein the task type includes a decentralized type and a centralized type; also used to send the subset of the first data that has not been uploaded to a third party to the third party when the task to be processed is of a centralized type, or to coordinate with the third party and other participants to perform decentralized computing through local task scheduling when the task to be processed is of a decentralized type; The second participant is used to: after detecting that the first participant has initiated a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant; and also to send the subset of the second data that has not been uploaded to the third party to the third party when the task to be processed is of a centralized type, or to coordinate the third party and other participants to perform distributed computing through local task scheduling when the task to be processed is of a decentralized type; A third party, the third party is used to: when the task to be processed is of a centralized type, combine the first data and the second data to perform centralized computing, or when the task to be processed is of a decentralized type, cooperate with other participants to perform decentralized computing through local computing.

2. The hybrid architecture system according to claim 1, characterized in that: The number of the second participants is multiple.

3. The hybrid architecture system according to claim 1, characterized in that: The first participant includes a first scheduling module, a first calculation module, a first communication module and a task type recommendation module, wherein the first communication module is used to communicate with the second participant and / or the third party; the task type recommendation module is used to initiate a task to be processed, determine first data associated with the task to be processed, and calculate a first time required to transmit a subset of the first data that has not been uploaded to the third party to the third party, and receive a second time from each other second participant, and determine the task type of the task to be processed based on the first time, the second time and a preset time threshold; The first scheduling module is used to schedule the first computing module to perform local calculations through local tasks to cooperate with a third party and other participants to perform distributed calculations when the task to be processed is of a distributed type, or to send a subset of the first data that has not been uploaded to the third party to the third party through the first communication module when the task to be processed is of a centralized type.

4. The hybrid architecture system according to claim 1, characterized in that: The second participant includes a second scheduling module, a second computing module and a second communication module. The second communication module is used to communicate with the first participant and / or the third party. The second scheduling module is used to: after detecting that the first participant initiates a task to be processed, determine the second data associated with the task to be processed, calculate the second time required to transmit the subset of the second data that has not been uploaded to the third party to the third party, and send the second time to the first participant through the second communication module, and when the task to be processed is a distributed type, schedule the second computing module through a local task to cooperate with the third party and other participants to perform distributed computing, or when the task to be processed is a centralized type, send the subset of the second data that has not been uploaded to the third party to the third party through the second communication module.

5. The hybrid architecture system according to claim 1, characterized in that: The third party includes a third scheduling module, a third computing module and a third communication module, and the third communication module is used to communicate with the first participant and / or the second participant; the third scheduling module is used to: when the task to be processed is of a centralized type, schedule the third computing module to perform centralized calculations in combination with the first data and the second data, or when the task to be processed is of a distributed type, schedule the third computing module to perform local calculations through local tasks to cooperate with other participants to perform distributed calculations.

6. A task scheduling method based on data transmission time consumption, characterized in that: Applied to any first participant in a participant subsystem that initiates a task to be processed, the participant subsystem further comprising at least one second participant associated with the task to be processed, the task scheduling method comprising: Submitting a task to be processed, determining first data associated with the task to be processed, and calculating a first time required to transmit a subset of the first data that has not been uploaded to a third party to the third party; receiving a second time required for transmitting a subset of the second data associated with the to-be-processed task from each other second participant and not uploaded to the third party to the third party; Determining a task type of the to-be-processed task based on the first time consumption, the second time consumption, and a preset time consumption threshold, wherein the task type includes a decentralized type and a centralized type; When the task to be processed is of a centralized type, a subset of the first data that has not been uploaded to the third party is sent to the third party, so that the third party can combine the first data and the second data uploaded to the third party after the second participant receives the task type to perform centralized calculations to obtain processing results; or when the task to be processed is of a distributed type, local task scheduling is used to cooperate with the third party and other participants to perform distributed calculations to obtain processing results.

7. The task scheduling method based on data transmission time consumption according to claim 6 is characterized in that: The step of determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold comprises: Selecting a maximum time from the first time and the second time as a first data transmission time; If the first data transmission time is greater than a preset time threshold, determining that the task type of the task to be processed is a distributed type; If the first data transmission time is less than or equal to a preset time threshold, it is determined that the task type of the to-be-processed task is a centralized type.

8. The task scheduling method based on data transmission time consumption according to claim 6 is characterized in that: The step of determining the task type of the to-be-processed task based on the first time consumption, the second time consumption and a preset time consumption threshold comprises: Obtaining task information of the task to be processed; Based on the task information, query from preset historical tasks a first historical task that matches the similarity of the task to be processed and whose task type is a decentralized type; Obtaining the intermediate transmission volume during the process of transmitting intermediate data between the first participant and other second participants in the process of processing the first historical task, and calculating the second data transmission time consumption based on the intermediate transmission volume and the current network status information, and using the second data transmission time consumption as the preset time consumption threshold; Selecting a maximum time from the first time and the second time as a first data transmission time; If the first data transmission time is greater than or equal to the preset time threshold, determining that the type of the task to be processed is a distributed type; If the first data transmission time consumption is less than the preset time consumption threshold, it is determined that the type of the task to be processed is a centralized type.

9. A task scheduling method based on data transmission time consumption, characterized in that: Applied to any second participant in a participant subsystem that is passively associated with a task to be processed, the participant subsystem also includes a first participant that initiates the task to be processed, and the task scheduling method includes: After detecting that the first participant has initiated a task to be processed, determining second data associated with the task to be processed, calculating a second time required to transmit a subset of the second data that has not been uploaded to the third party to the third party, and sending the second time to the first participant; receiving the task type of the to-be-processed task returned by the first participant, and sending the subset of the second data that has not been uploaded to the third party to the third party when the to-be-processed task is of a centralized type, so that the third party can perform centralized computing on the basis of the second data and the first data uploaded to the third party by the first participant based on the task type to obtain a processing result; or performing decentralized computing in coordination with the third party and other participants through local task scheduling to obtain a processing result when the to-be-processed task is of a decentralized type; The task type is determined by the first participant according to the first time required for the first participant to transmit the subset of the first data associated with the task to be processed that has not been uploaded to the third party to the third party, the second time of each other second participant, and a preset time threshold; The task types include decentralized types and centralized types.

10. The task scheduling method based on data transmission time consumption according to claim 6 or 9, characterized in that: The task scheduling method further includes: When in an idle state, the local first data set is uploaded to the third party.

11. The task scheduling method based on data transmission time consumption according to claim 6 or 9, characterized in that: When the task to be processed is of a distributed type, the steps of obtaining the processing result by cooperating with the third party and other participants to perform distributed computing through local task scheduling include: If the decentralized type of the task to be processed is a horizontal federation type, sending first intermediate data to the third party; wherein the first intermediate data is generated by each participant based on a corresponding first model gradient, and the first model gradient is calculated by each participant based on a corresponding local model; Receiving aggregated intermediate data sent by a third party, wherein the aggregated intermediate data is obtained by the third party aggregating the first intermediate data; updating model parameters based on the aggregated intermediate data to obtain updated local models, and continuing to generate new first intermediate data based on the respective updated local models; Iterate and return to the step of sending the first intermediate data to the third party until the first preset completion condition is met, and use the data result corresponding to the first preset completion condition as the processing result.

12. The task scheduling method based on data transmission time consumption according to claim 6 or 9, characterized in that: When the task to be processed is of a distributed type, the steps of obtaining the processing result by cooperating with the third party and other participants to perform distributed computing through local task scheduling include: If the decentralized type of the task to be processed is a vertical federation type, encrypting and masking the second intermediate data to obtain an intermediate result, and sending the intermediate result to the third party; The second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model; Receiving the decrypted intermediate result sent by the third party after decrypting the intermediate result, updating the respective model parameters based on the decrypted intermediate result, obtaining the respective updated local models, and continuing to generate new intermediate results based on the respective updated local models; The process iterates back to the step of encrypting the second intermediate data and performing mask processing to obtain an intermediate result until a second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result.

13. A task scheduling method based on data transmission time consumption, characterized in that: Applied to a third party, the task scheduling method includes: When the task type of the task to be processed initiated by the first participant is a centralized type, a centralized operation is performed on the first data and the second data to obtain a processing result, wherein the first data is the data associated with the task to be processed uploaded by the first participant to the third party, and the second data is the data associated with the task to be processed uploaded by each second participant to the third party; or When the task type of the task to be processed initiated by the first participant is a distributed type, the processing result is obtained by cooperating with other participants through local calculation to perform distributed calculation; Among them, the task type of the pending task initiated by the first participant is determined by the first participant based on the first time required for the first participant to transmit a subset of the first data of the pending task that has not been uploaded to the third party to the third party, the second time required for each other second participant to transmit a subset of the second data of the pending task that has not been uploaded to the third party to the third party, and a preset time threshold.

14. The task scheduling method based on data transmission time consumption according to claim 13, characterized in that: When the task type of the to-be-processed task initiated by the first participant is a centralized type, a centralized operation is performed in combination with the first data and the second data to obtain a processing result, including: When the task type of the task to be processed initiated by the first participant is a centralized type, the first data and the second data are combined, the centralized task is scheduled by adopting a centralized scheduling mode, and the first participant and each second participant perform privacy calculation on the task to be processed according to the target scheduling mode corresponding to the task type to obtain a processing result.

15. The task scheduling method based on data transmission time consumption according to claim 14, characterized in that: The step of combining the first data and the second data, scheduling the centralized task by adopting a centralized scheduling mode, and cooperating with the first participant and each second participant to perform privacy calculation on the task to be processed according to the target scheduling mode corresponding to the task type to obtain a first processing result includes: Determine whether the complete set of first data of the first participant and the complete set of second data of each second participant are stored locally; If not, scheduling the first participant to encrypt and upload the subset of the first data that has not been uploaded to the third party, and scheduling other second participants to encrypt and upload the subset of the second data that has not been uploaded to the third party, and storing the full set of the first data and the full set of the second data; The entire set of the first data and the entire set of the second data are retrieved locally, and the tasks to be processed are centrally processed to obtain processing results.

16. The task scheduling method based on data transmission time consumption according to claim 13, characterized in that: When the task type of the task to be processed initiated by the first participant is a distributed type, local calculation is performed to cooperate with other participants to perform distributed calculations to obtain processing results, including: When the task type of the task to be processed initiated by the first participant is a decentralized type, the decentralized type tasks are scheduled by adopting a decentralized scheduling mode, and the first participant and the second participant perform privacy calculations on the task to be processed according to the target scheduling mode corresponding to the task type, and obtain processing results.

17. The task scheduling method based on data transmission time consumption according to claim 16, characterized in that: The method of scheduling the distributed type tasks by adopting a distributed scheduling mode, and cooperating with the first participant and the second participant to perform privacy calculation on the to-be-processed tasks according to the target scheduling mode corresponding to the task type, and obtaining the processing result, includes: If the decentralized type of the task to be processed is a horizontal federated decentralized type, receiving first intermediate data encrypted and sent by each participant; The first intermediate data is generated by each participant based on the corresponding first model gradient, and the first model gradient is obtained by each participant performing privacy calculation based on the corresponding local model; aggregating the first intermediate data to obtain aggregated intermediate data; Sending the aggregated intermediate data to each participant, so that each participant updates their own model parameters based on the aggregated intermediate data, obtains their own updated local models, and performs privacy calculations based on their own updated local models to continue to generate new first intermediate data; Iterate and return to the step of receiving the first intermediate data sent by each participant until the first preset completion condition is met, and take the data result corresponding to the first preset completion condition as the processing result.

18. The task scheduling method based on data transmission time consumption according to claim 16, characterized in that: The method of scheduling the distributed type tasks by adopting a distributed scheduling mode, and cooperating with the first participant and the second participant to perform privacy calculation on the to-be-processed tasks according to the target scheduling mode corresponding to the task type, and obtaining the processing result, includes: If the decentralized type of the task to be processed is a vertical federation type, receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data; The second intermediate data is generated by each participant based on the corresponding encrypted second model gradient exchange, and the second model gradient is calculated by each participant based on the corresponding local model; Decrypting the intermediate result, and sending the decrypted intermediate result to the corresponding participant, so that each participant can update their own model parameters based on the intermediate result, obtain their own updated local models, and continue to generate new intermediate results based on their own updated local models; Iterate and return to the step of receiving the intermediate results sent by each participant after encrypting and masking the second intermediate data, until the second preset completion condition is met, and the data result corresponding to the second preset completion condition is used as the processing result.

19. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a program stored in the memory for implementing the task scheduling method. The memory is used to store a program for implementing a task scheduling method based on time-consuming data transmission; The processor is used to execute a program for implementing the task scheduling method to implement the steps of the task scheduling method as claimed in any one of claims 6 to 18.

20. A storage medium, characterized in that: The storage medium stores a program for implementing a task scheduling method based on data transmission time consumption, and the program for implementing a task scheduling method based on data transmission time consumption is executed by a processor to implement the steps of the task scheduling method as described in any one of claims 6 to 18.

21. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the task scheduling method based on data transmission time consumption as claimed in any one of claims 6 to 18 are implemented.

Images