Task processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

This application provides a task processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. The method includes: acquiring a task dataset of a task to be processed and pre-configured parameters for the task to be processed; dividing the task dataset of the task to be processed into multiple data buckets based on a pre-configured number of buckets in the pre-configured parameters; each data bucket including at least one task data of the task to be processed; dividing the task to be processed into multiple sub-tasks to be executed based on pre-configured model parameters in the pre-configured parameters; allocating a target node to each data bucket from a node cluster; wherein each data bucket is allocated to one target node; and processing the multiple sub-tasks to be executed based on the task data in each data bucket through the target node. This application can improve the processing efficiency of the task to be processed and the utilization rate of the computing resources of the target node.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a task processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Data is the foundation of datafication, networking, and intelligence, and it contains significant value. With data emerging as a new production factor and the enactment of data security laws, companies are placing greater emphasis on data privacy protection. Multiple companies are continuously promoting the construction of data collaborative modeling platforms while adhering to legal compliance. On the one hand, they are accelerating the execution performance of large-scale tasks by breaking down tasks through distributed scheduling frameworks; on the other hand, in business collaborations, they are safeguarding the data assets of various companies through technologies such as federated learning and multi-party secure computation. Data privacy protection involves multi-party collaborative modeling scenarios, which can be further subdivided into horizontal federated learning and vertical federated learning. The main difference between the two lies in the division of features in the sample space.

[0003] In multi-party collaboration scenarios, different federated modeling approaches can be selected based on the specific business collaboration context and the degree of feature overlap. For collaborations between businesses in similar industries, a horizontal federated learning approach can be used to model user profiles, achieving an effect similar to centralized training while ensuring data compliance by remaining locally. For collaborations between businesses in different industries, a vertical federated learning approach can be used, achieving secure multi-party modeling through encrypted gradient information transmission, thereby ensuring data security for all parties while maximizing business benefits. However, the federated scheduling systems used in these technologies suffer from low processing efficiency, unbalanced allocation of computing resources, and poor utilization of computing resources when supporting business collaborations across different industries and scenarios. Summary of the Invention

[0004] This application provides a task processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the processing efficiency of the task to be processed and the utilization rate of the computing resources of the target node.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a task processing method, the method comprising: acquiring a task dataset of a task to be processed and pre-configured parameters for the task to be processed; dividing the task dataset of the task to be processed into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters; each data bucket including at least one task data of the task to be processed; dividing the task to be processed into multiple sub-tasks to be executed based on pre-configured model parameters in the pre-configured parameters; allocating a target node to each data bucket from a node cluster; wherein each data bucket is allocated to a target node; and processing the multiple sub-tasks to be executed based on the task data in each data bucket through the target node.

[0007] This application provides a task processing apparatus, comprising: a data acquisition module for acquiring a task dataset of a task to be processed and pre-configured parameters for the task to be processed; a data bucketing module for dividing the task dataset of the task to be processed into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters; each data bucket including at least one task data of the task to be processed; a task partitioning module for dividing the task to be processed into multiple sub-tasks to be executed based on pre-configured model parameters in the pre-configured parameters; a node allocation module for allocating a target node to each data bucket from a node cluster; wherein each data bucket is allocated to a target node; and a task processing module for processing the multiple sub-tasks to be executed based on the task data in each data bucket through the target node.

[0008] In the above scheme, the node allocation module is further configured to determine the number of nodes in the node cluster that are currently in an idle state; in response to the number of nodes in an idle state being greater than or equal to the number of pre-configured buckets, to randomly allocate a target node to each data bucket from the nodes in the idle state; wherein each data bucket is allocated to a different target node.

[0009] In the above scheme, the node allocation module is further configured to determine the number of nodes in the node cluster that are currently in an idle state; in response to the number of nodes in an idle state being less than the number of pre-configured buckets, to allocate a target node to each data bucket from the nodes in the idle state; wherein at least two data buckets are allocated to the same target node.

[0010] In the above scheme, the node allocation module is further configured to: obtain the resource release amount of nodes in the node cluster that are in a non-idle state; generate a new node based on the resource release amount in response to the resource release amount being greater than or equal to the amount of computing resources required for a subtask to be executed; filter target data buckets from at least two data buckets allocated to the same target node; and allocate the target data buckets to the new node so that the new node can perform task processing on at least one subtask to be executed in the task to be processed based on the task data in the target data bucket.

[0011] In the above scheme, the task processing module is further configured to determine the amount of computing resources of the target node; based on the amount of computing resources, determine a threshold for the number of subtasks to be processed by the target node in each round of task processing; obtain the priority of each subtask to be processed; and based on the number threshold and the task data in the data bucket allocated to the target node, perform at least one round of task processing on the at least one subtask to be processed according to the priority.

[0012] In the above scheme, the device further includes a data transmission module, used to acquire the data type of the response data obtained after the task processing; determine the target receiving end of the response data based on the data type; wherein, the task to be processed is a multi-terminal collaborative task, and the multi-terminal collaborative task is processed by multiple task processing ends respectively; the target receiving end is at least one of the multiple task processing ends; and the response data is asynchronously transmitted to the target receiving end through a preset communication channel.

[0013] In the above scheme, the task division module is further used to divide the task to be processed into multiple module tasks based on the pre-configured model parameters in the pre-configured parameters; divide each module task into multiple sub-tasks; and determine each of the sub-tasks as the sub-task to be executed.

[0014] In the above scheme, the task division module is further used to divide the task to be processed into multiple module tasks based on the pre-configured model parameters in the pre-configured parameters; divide each module task into multiple sub-tasks; obtain the sub-tasks at the same task level in each module task to form a sub-task set; and determine all the sub-tasks in the sub-task set as a sub-task to be executed.

[0015] In the above scheme, the data bucketing module is further used to perform hash processing on the task data in the task dataset to obtain the hash value of the task data; according to the hash value of the task data, the task data in the task dataset is allocated to a pre-configured number of buckets to obtain multiple data buckets.

[0016] This application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the task processing method provided in this application.

[0017] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the task processing method provided in this application when executed by a processor.

[0018] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the task processing method provided in this application.

[0019] The embodiments of this application have the following beneficial effects:

[0020] First, the task dataset and pre-configured parameters for the task to be processed are obtained. Then, according to the pre-configured number of buckets in the pre-configured parameters, the task dataset is divided into multiple data buckets. Each data bucket contains at least one task data item from the task to be processed. The bucketed data can be processed in parallel, improving data processing speed and thus increasing the processing speed of the task. Next, based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed. Then, a target node is assigned to each data bucket from the node cluster. Finally, through the target node, the multiple sub-tasks to be executed are processed based on the task data in each data bucket. In this way, data can be bucketed first, reducing the amount of data processed by each node, and then the processing task can be decomposed into multiple sub-tasks, significantly improving the processing efficiency of the task. Furthermore, after decomposing the task into multiple sub-tasks, computing resources can be rationally allocated according to the computing resource requirements of each sub-task, avoiding overloading some nodes while leaving others idle, and improving the flexibility and utilization of computing resources on the target nodes. Attached Figure Description

[0021] Figure 1 This is an optional architecture diagram of the task processing system provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0023] Figure 3 This is an optional flowchart illustrating the task processing method provided in the embodiments of this application;

[0024] Figure 4This is another optional flowchart illustrating the task processing method provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram illustrating the implementation process of data bucket partitioning provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of an implementation process for dividing the subtasks to be executed according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of another implementation process for dividing the subtasks to be executed provided in the embodiments of this application;

[0028] Figure 8 This is a schematic diagram illustrating the implementation flow of processing multiple sub-tasks to be executed provided in the embodiments of this application;

[0029] Figure 9 This is another optional flowchart illustrating the task processing method provided in the embodiments of this application;

[0030] Figure 10 This is an optional schematic diagram of the federated scheduling system provided in the embodiments of this application;

[0031] Figure 11 This is another optional schematic diagram of the federated scheduling system provided in the embodiments of this application;

[0032] Figure 12 This is another optional schematic diagram of the federated scheduling system provided in the embodiments of this application;

[0033] Figure 13 This is another optional architecture diagram of the task processing system provided in the embodiments of this application;

[0034] Figure 14 This is a schematic diagram of an optional architecture for distributed asynchronous transformation provided in the embodiments of this application;

[0035] Figure 15 This is a schematic diagram of an optional architecture for data transmission provided in an embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0041] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0042] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0043] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0044] 2) Human-computer interaction interface, which is used to provide human-computer interaction functions / display task information.

[0045] For example, graphical user interfaces (GUIs) include augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing users to see stereoscopic images without wearing special glasses), multimodal interfaces (interfaces that combine multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interfaces (BMIs).

[0046] The federated learning schemes used in related technologies are usually complex in terms of syntax and network configuration, and cannot control dynamic scaling at a fine-grained level. They also tend to consume too many ports in the algorithm implementation, resulting in a certain degree of resource waste. In addition, they have limited support for deep learning and the scheduling schemes are inflexible.

[0047] To address at least one of the aforementioned problems with methods in related technologies, embodiments of this application provide a task processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the processing efficiency of tasks to be processed and the utilization rate of target node computing resources. First, the task dataset and pre-configured parameters for the tasks to be processed are obtained. Then, according to the number of pre-configured buckets in the pre-configured parameters, the task dataset is divided into multiple data buckets, each containing at least one task data item for the task to be processed. The bucketed data can be processed in parallel, improving data processing speed and thus increasing the processing speed of the task to be processed. Next, based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed. Then, a target node is allocated to each data bucket from the node cluster. Finally, through the target node, the multiple sub-tasks to be executed are processed based on the task data in each data bucket. In this way, the processing efficiency of the task can be significantly improved by decomposing the task into multiple subtasks. Furthermore, after decomposing the task into multiple subtasks, computing resources can be allocated reasonably according to the computing resource requirements of each subtask, avoiding overloading of computing resources on some nodes while leaving computing resources on other nodes idle, thereby improving the flexibility and utilization of computing resources allocation on the target node.

[0048] Here, we first describe an exemplary application of the task processing device in the embodiments of this application. This task processing device is an electronic device used to implement a task processing method. In one implementation, the task processing device (i.e., electronic device) provided in the embodiments of this application can be implemented as a terminal or as a server. In one implementation, the task processing device provided in the embodiments of this application can be implemented as any terminal with data processing and task processing functions, such as a laptop, tablet, desktop computer, mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, intelligent robot, smart home appliance, and smart vehicle device. In another implementation, the task processing device provided in the embodiments of this application can also be implemented as a server, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of this application. The following describes an exemplary application when the task processing device is implemented as a server.

[0049] See Figure 1 , Figure 1 This is an optional architecture diagram of the task processing system provided in this application embodiment. The task processing system 10 in this application embodiment includes at least a terminal 100, a network 200, and a server 300. A task processing application is installed on the terminal 100. The task processing application can provide task processing functions. The task processing functions can be to detect the system performance of the server under any indicator, such as the system performance of CPU utilization, disk utilization, etc., or the performance of the terminal under any indicator.

[0050] The following description uses the installation of a task processing application on a terminal as an example. In this embodiment, server 300 can be a server for the task processing application. Server 300 can constitute the task processing device of this embodiment, that is, the task processing method of this embodiment is implemented through server 300. Terminal 100 connects to server 300 through network 200, which can be a wide area network or a local area network, or a combination of both.

[0051] See Figure 1When a user wants to perform task processing, they can input a task processing operation in the client of the task processing application. This operation is used to select a task to be processed or to input a task identifier for the task. Then, terminal 100 encapsulates the task identifier into a task processing request and sends the request to server 300 via network 200. Server 300 then responds to the task processing request, obtaining the task dataset and pre-configured parameters for the task. Next, based on the number of pre-configured buckets in the pre-configured parameters, server 300 divides the task dataset into multiple data buckets. Then, based on the pre-configured model parameters, server 300 divides the task into multiple sub-tasks to be executed. Next, server 300 allocates a target node to each data bucket from the node cluster. Finally, server 300 processes the multiple sub-tasks through the target nodes, based on the task data in each data bucket. After obtaining the task processing result, server 300 sends the result to terminal 100, which can then display the result on the current interface of the task processing application.

[0052] In some embodiments, the implementation steps of the task processing method can also be executed by the terminal 100. That is, after determining the task to be processed, the terminal 100 obtains the task dataset and pre-configured parameters for the task to be processed; then, the terminal 100 divides the task dataset into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters; then, the terminal 100 divides the task to be processed into multiple sub-tasks to be executed based on the pre-configured model parameters in the pre-configured parameters; then, the terminal 100 allocates a target node to each data bucket from the node cluster; finally, the terminal 100 processes the multiple sub-tasks to be executed through the target nodes based on the task data in each data bucket.

[0053] The task processing method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server. The task dataset and pre-configured parameters for the task to be processed can be obtained through the cloud server. Alternatively, the task dataset can be divided into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters. Alternatively, the task can be divided into multiple sub-tasks to be executed based on the pre-configured model parameters in the pre-configured parameters. Alternatively, a target node can be allocated to each data bucket from the node cluster through the cloud server. Alternatively, the task can be processed by the cloud server through the target node based on the task data in each data bucket.

[0054] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device 400 may be a task processing device, which includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0055] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0056] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0057] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0058] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0059] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0060] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 452 for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.; presentation module 453 for enabling the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., display screen, speaker, etc.); input processing module 454 for detecting and translating one or more user inputs or interactions from one or more input devices 432.

[0061] In some embodiments, the task processing apparatus provided in this application can be implemented in software. Figure 2 A task processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data acquisition module 4551, a data bucketing module 4552, a task partitioning module 4553, a node allocation module 4554, and a task processing module 4555. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0062] In other embodiments, the task processing device provided in this application can be implemented in hardware. As an example, the task processing device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the task processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0063] The task processing methods provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the task processing methods in the embodiments of this application can be executed by a server, by a terminal, or by interaction between a server and a terminal.

[0064] Figure 3 This is an optional flowchart illustrating the task processing method provided in the embodiments of this application. The following will be combined with... Figure 3 The steps shown are explained as follows: Figure 3 As shown, taking the server as the execution subject of the task processing method as an example, the method includes the following steps S101 to S105:

[0065] Step S101: Obtain the task dataset and pre-configured parameters for the task to be processed.

[0066] In this embodiment, the task to be processed can be a task for training a model, a task for improving the model's performance, etc. Under the federated learning framework, the task to be processed can be a specific machine learning or data processing task completed through distributed computing and collaboration. The specific task to be processed can be set according to actual circumstances, and this embodiment does not limit this. For example, different financial institutions each possess their own user transaction data and can jointly train an anomaly detection model to identify potential abnormal behavior; the task to be processed could be the task of training an anomaly detection model. The task dataset is a dataset composed of the sample data required to process the task to be processed. For example, if the task to be processed is the task of training an anomaly detection model, the task dataset could be a dataset composed of user transaction data from financial institutions. Pre-configured parameters can include the number of pre-configured buckets and pre-configured model parameters. Under the federated learning framework, the pre-configured parameters can be preset in a pre-configured configuration file, and can be obtained by parsing the pre-configured configuration file.

[0067] Step S102: Based on the number of pre-configured buckets in the pre-configured parameters, divide the task dataset of the task to be processed into multiple data buckets.

[0068] Here, each data bucket contains at least one task data for the task to be processed.

[0069] In this embodiment, the number of pre-configured buckets can be set according to the actual amount of data in the task dataset, and this embodiment does not limit this. Each data bucket includes at least one task data of the task to be processed, and each data bucket can be a subset of the dataset consisting of at least one task data. The bucketing technique for dividing the task dataset of the task to be processed into multiple data buckets can include hash bucketing, range bucketing, and random bucketing, etc., and this embodiment does not limit this. Assuming a large-scale data processing task to clean and analyze a dataset containing 10 million records, with a pre-configured number of buckets of 10, the dataset can be divided into 10 data buckets using hash bucketing, with each data bucket containing 1 million records.

[0070] Step S103: Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed.

[0071] In this embodiment, the pre-configured model parameters can be set according to actual conditions, and this embodiment does not limit this. The pre-configured model parameters may include model structure parameters and model training parameters. Model structure parameters define the structure and characteristics of the model, ensuring that all participants use a consistent model architecture for training within the federated learning framework. Model structure parameters may include model type, number of model layers, number of nodes per layer, activation function, and output layer activation function, etc. Model training parameters may include learning rate, number of training epochs, number of iterations per epoch, batch size, optimizer, loss function, and evaluation metrics, etc. Based on the pre-configured model parameters, independently executable computational tasks can be identified within the task to be processed, and these independently executable computational tasks are designated as subtasks to be executed.

[0072] Taking the reinforcement learning framework (Ray) engine as an example, the Ray engine can divide a task into multiple independent tasks or stateful objects (Actors) through efficient parallel computing and distributed task scheduling, thereby improving processing efficiency. For example, a big data processing task can be divided into multiple sub-tasks to be executed, such as data reading, data processing, and data storage. Then, the Ray "@ray.remote" decorator can be used to define Tasks, defining each sub-task as a remote function. Tasks are stateless functions, suitable for performing independent computational tasks, such as data preprocessing (converting raw data into a format suitable for model training, including standardization, normalization, and outlier removal) or model evaluation. Actors are stateful objects, suitable for tasks that require maintaining state, such as the model training process. The model training logic can be encapsulated in a class and defined as an Actor using the "@ray.remote" decorator. Depending on the model's complexity and resource requirements, multiple Actor instances can be created for parallel training. The number of Actors to be processed in parallel can be determined based on the model's parameters (such as the number of layers and nodes). For example, for large models, different parts of the model can be assigned to different Actors. The resource requirements for Tasks and Actors can be set based on the resource needs of model training (such as the number of Central Processing Units (CPUs) and Graphics Processing Units (GPUs)).

[0073] Step S104: Assign a target node to each data bucket from the node cluster.

[0074] Here, each data bucket is assigned to a target node.

[0075] In this embodiment, a node can refer to an independent computing unit, such as a physical server, virtual machine, or container. Each node can possess certain computing power, memory, and storage resources, enabling it to execute specific computing tasks. Each node can be allocated specific resources according to task requirements, such as the number of CPU cores, memory size, and number of GPUs. Nodes can perform various computing tasks, such as data processing, model training, and data analysis. Nodes can communicate with each other through a network, exchanging data and information. A node cluster refers to a collection of multiple nodes connected together through a network, working collaboratively to complete complex computing tasks. Node clusters can provide powerful computing capabilities, suitable for handling large-scale data and complex tasks.

[0076] The idle nodes in the node cluster can be identified, and then each data bucket can be assigned an idle node as the target node. For example, if there are 10 data buckets and 10 nodes in the node cluster, all of which are idle, target node 1 processes the data in data bucket 1, target node 2 processes the data in data bucket 2, and so on. Alternatively, if there are 10 data buckets and 10 nodes in the node cluster, but only 3 nodes are idle, target node 1 processes the data in data bucket 1, target node 2 processes the data in data bucket 2, target node 3 processes the data in data bucket 3, target node 1 processes the data in data bucket 4, target node 2 processes the data in data bucket 5, and so on.

[0077] Step S105: Based on the task data in each data bucket, process multiple subtasks to be executed through the target node.

[0078] In this embodiment, the target node is responsible for processing the task data in the assigned data bucket. The task data can be the input to the sub-task to be executed, used to perform specific processing logic. Assume a large-scale data processing task to clean and analyze a dataset containing 10 million records. Using the Ray engine for distributed processing, the dataset is divided into 10 data buckets, each containing 1 million records. Each node is responsible for processing one data bucket. Using Ray's resource management function, each data bucket is assigned to a target node. The processing logic for each sub-task is defined. For example, each sub-task is responsible for cleaning the data in the data bucket. On each target node, based on the task data in the assigned data bucket, the processing logic of the sub-task is executed to clean the data in the data bucket.

[0079] In this embodiment, firstly, the task dataset and pre-configured parameters for the task to be processed are obtained. Then, according to the pre-configured number of buckets in the pre-configured parameters, the task dataset is divided into multiple data buckets. Each data bucket contains at least one task data item for the task to be processed. The bucketed data can be processed in parallel, improving data processing speed and thus increasing the processing speed of the task. Next, based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed. Then, a target node is allocated to each data bucket from the node cluster. Finally, through the target node, the multiple sub-tasks to be executed are processed based on the task data in each data bucket. In this way, by decomposing the processing task into multiple sub-tasks, the processing efficiency of the task to be processed can be significantly improved. Furthermore, after decomposing the task into multiple sub-tasks, computing resources can be rationally allocated according to the computing resource requirements of each sub-task, avoiding overload of computing resources on some nodes while other nodes are idle, thus improving the flexibility and utilization of computing resources allocated to the target node.

[0080] The following examples illustrate the application scenarios of the task processing method provided in the embodiments of this application.

[0081] In the field of financial risk control, federated learning provides strong technical support for building joint anomaly detection models. Multiple banks and financial institutions, while strictly adhering to data privacy regulations, can use federated learning technology to jointly analyze their respective customer data without sharing specific customer information. The task processing method provided in this application embodiment can be applied to various participating parties. For example, a bank can collaborate with a telecommunications operator, using the bank's customer financial transaction records and the operator's communication behavior data to jointly build a corporate financing risk control model. The task processing method provided in this application embodiment can be applied to both the bank's side and the telecommunications operator's side.

[0082] The field of smart healthcare has extremely high requirements for data privacy and security. Federated learning technology provides a secure and efficient solution for the sharing and utilization of medical data. Through federated learning, multiple hospitals can collaborate, integrating their respective datasets to train a federated model that incorporates knowledge from multiple parties. The task processing method provided in this application can be applied to each participating party. In practical applications, such as when performing patient DNA sequencing analysis, the DNA databases of each hospital and the patient's DNA sequence remain mutually unknown, yet the federated learning model can still provide patients with accurate disease diagnosis and prevention predictions. The task processing method provided in this application can be applied to each hospital individually.

[0083] The task processing method of this application embodiment will be described below in conjunction with the above scenario.

[0084] Figure 4 This is another optional flowchart illustrating the task processing method provided in the embodiments of this application, such as... Figure 4 As shown, the method includes the following steps S201 to S212:

[0085] Step S201: The terminal receives the task processing operation input by the user.

[0086] Task processing operations include selection operations or input operations. The selection operation is used to select the task to be processed, or the input operation is used to input the task identifier of the task to be processed.

[0087] In step S202, the terminal encapsulates the task identifier into the task processing request.

[0088] A task processing request is used to request the server to process the task to be processed.

[0089] Step S203: The terminal sends a task processing request to the server.

[0090] In this embodiment of the application, the terminal sends a task processing request to the server to request the server to process the task to be processed.

[0091] In step S204, the server responds to the task processing request by obtaining the task dataset and pre-configured parameters for the task to be processed.

[0092] In this embodiment, the server receives a task processing request sent by the terminal and parses the request to obtain a pre-configuration file. Further parsing of the pre-configuration file yields the task dataset and pre-configuration parameters for the task to be processed. The specific implementation of obtaining the task dataset and pre-configuration parameters for the task can be found in step S101, and will not be elaborated here.

[0093] In step S205, the server divides the task dataset of the task to be processed into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters.

[0094] In some embodiments, see Figure 5 , Figure 5 Step S205 can be achieved through the following steps S2051 to S2052:

[0095] Step S2051: Hash the task data in the task dataset to obtain the hash value of the task data.

[0096] In this embodiment, hashing is a process of converting data of arbitrary length into a hash value of fixed length. The hash value is typically a number or string. Each task data in the task dataset is hashed to obtain its hash value. For example, a task dataset contains the following task data: ["data1", "data2", "data3", "data4", "data5", "data6", "data7", "data8", "data9", "data10"]. The hash function `hash` is used to calculate the hash value for each task data. Assume the resulting hash values ​​are as follows: [12345, 67890, 23456, 34567, 45678, 56789, 67890, 78901, 89012, 90123].

[0097] Step S2052: Based on the hash value of the task data, distribute the task data in the task dataset into buckets of a pre-configured number of buckets to obtain multiple data buckets.

[0098] In this embodiment, the hash value of the task data can be moduloed according to the number of pre-configured buckets to obtain the modulo result. Based on the modulo result, the task data in the task dataset is allocated to the number of buckets pre-configured, resulting in multiple data buckets. The modulo operation is a mathematical operation used to calculate the remainder after dividing two integers. The hash value of the task data can be divided by the number of pre-configured buckets to obtain the remainder. According to the relationship between the remainder and the pre-configured buckets (e.g., numbering the pre-configured buckets according to the number of pre-configured buckets), the task data is mapped to the pre-configured buckets. Assume that 5 data buckets are pre-configured, i.e., the number of buckets is 5. The modulo operation on the hash value maps the task data to the 5 pre-configured data buckets. Suppose the hash values ​​of the task data ["data1", "data2", "data3", "data4", "data5", "data6", "data7", "data8", "data9", "data10"] are [20, 21, 22, 23, 24, 25, 26, 27, 28, 29], and the modulo result is [0, 1, 2, 3, 4, 0, 1, 2, 3, 4]. The task data will be allocated to data buckets as follows: bucket0: ["data1", "data6"]; bucket1: ["data2", "data7"]; bucket2: ["data3", "data8"]; bucket3: ["data4", "data9"]; bucket4: ["data5", "data10"].

[0099] Through steps S2051 to S2052, hash processing can be used to evenly distribute task data into multiple pre-configured data buckets, which can ensure the uniformity of data distribution and is suitable for large-scale data processing and distributed computing scenarios. The data after bucketing can be processed in parallel, which improves the data processing speed and thus improves the processing speed of the task to be processed.

[0100] Step S206: The server divides the task to be processed into multiple sub-tasks to be executed based on the pre-configured model parameters in the pre-configured parameters.

[0101] In some embodiments, see Figure 6 , Figure 6 This shows that step S206 can be achieved through the following steps S2061A to S2063A:

[0102] Step S2061A: Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple module tasks.

[0103] In this embodiment, pre-configured model parameters refer to model-related parameters defined before the task begins. These parameters may include the model's structure (e.g., number of layers, number of nodes), training parameters (e.g., learning rate, batch size), and other settings related to model operation. A module task refers to an independent task unit divided according to the pre-configured model parameters. For example, suppose a large-scale machine learning task is being processed, and the goal of the task is to train a deep learning model to process user transaction data. The task is divided into three independent modules: reading, sample intersection, and model training. The pre-configured model parameters include: a 5-layer neural network, 1 million transaction records divided into 10 categories, a learning rate of 0.001, a batch size of 32, and 10 training epochs. The task can be divided into three module tasks: reading, sample intersection, and model training.

[0104] Step S2062A: Divide each module task into multiple subtasks.

[0105] In this embodiment, each module task can be further subdivided into multiple subtasks according to the pre-configured model parameters and the execution logic of each module task. For example, module task 1, the reading task, can be divided into three subtasks: subtask 1.1 (reading transaction records from the database); subtask 1.2 (sorting transaction records by date); and subtask 1.3 (dividing the sorted data into training set, validation set, and test set). Module task 2, the sample intersection task, can be divided into three subtasks: subtask 2.1 (extracting positive samples from the training set); subtask 2.2 (extracting negative samples from the training set); and subtask 2.3 (performing an intersection operation on positive and negative samples to ensure sample balance). Module Task 3, model training, can be divided into 5 sub-tasks, including: Sub-task 3.1 (initializing the neural network model); Sub-task 3.2 (training the first round using the training set); Sub-task 3.3 (evaluating the model using the validation set); Sub-task 3.4 (adjusting the model parameters based on the evaluation results); Sub-task 3.5 (repeating the above steps for multiple rounds of training until the stopping condition is met).

[0106] Step S2063A: Identify each subtask as a subtask to be executed.

[0107] In this embodiment, the subtask to be executed refers to a specific task that is ready and can be allocated to computing resources for processing. After the subtask to be executed is determined, the computing node can execute it. Each subtask is determined as a subtask to be executed and can be allocated to computing resources for processing. For example: Subtask 1.1 (reading transaction records from the database) can be allocated to a CPU node. Subtask 1.2 (sorting transaction records by date) can be allocated to another CPU node. Subtask 2.1 (extracting positive samples from the training set) can be allocated to a GPU node. Subtask 2.2 (extracting negative samples from the training set) can be allocated to another GPU node. Subtask 3.1 (initializing the neural network model) can be allocated to a GPU node. Subtask 3.2 (performing the first round of training using the training set) can be allocated to a GPU node. Subtask 3.3 (evaluating the model using the validation set) can be allocated to a CPU node. Subtask 3.4 (adjusting model parameters based on the evaluation results) can be allocated to a CPU node.

[0108] By using steps S2061A to S2063A, the task to be processed can be divided into multiple module tasks, further subdivided into multiple sub-tasks, and resources can be allocated, which can significantly improve task management efficiency and resource utilization.

[0109] In some embodiments, see Figure 7 , Figure 7 Step S206 can be achieved through the following steps S2061B to S2064B:

[0110] Step S2061B: Based on the pre-configured model parameters in the pre-configured parameters, divide the task to be processed into multiple module tasks.

[0111] It should be noted that step S2061B is the same as S2061A above, and the implementation details of step S2061B in this embodiment will not be repeated.

[0112] Step S2062B: Divide each module task into multiple subtasks.

[0113] In this embodiment of the application, it should be noted that step S2062B is the same as S2062A above, and the implementation details of step S2062B will not be repeated in this embodiment of the application.

[0114] Step S2063B: Obtain subtasks at the same task level in each module task to form a subtask set.

[0115] In this embodiment, the level of each subtask in each module task is determined, and then the subtasks in each module task at the same task level are formed into a subtask set. For example, module task 1 includes 3 subtasks: subtask 1.1; subtask 1.2; subtask 1.3. Module task 2 includes 3 subtasks: subtask 2.1; subtask 2.2; subtask 2.3. Module task 3 includes 5 subtasks: subtask 2.1; subtask 3.2; subtask 3.3; subtask 3.4; subtask 3.5. Subtask 1.1, subtask 2.1, and subtask 3.1 can be considered as a subtask set 1; subtask 1.2, subtask 2.2, and subtask 3.2 can be considered as a subtask set 2; and subtask 1.3, subtask 2.3, and subtask 3.3 can be considered as a subtask set 3.

[0116] Step S2064B: Determine all subtasks in the subtask set as a single subtask to be executed.

[0117] In this embodiment, all subtasks in each subtask set are identified as subtasks to be executed and can be allocated to computing resources for processing. For example, subtask set 1 can be allocated to a CPU node. Subtask set 2 can be allocated to another CPU node. Subtask set 3 can be allocated to a GPU node.

[0118] Through steps S2061B to S2064B, subtasks at the same task level in each module task can be divided into subtask sets, and all subtasks in these subtask sets can be identified as a subtask to be executed. This enables finer-grained task management and resource allocation, making task execution more efficient and fully utilizing computing resources.

[0119] Step S207: The server determines the number of nodes in the node cluster that are currently in an idle state.

[0120] In this embodiment, the running status of each node in the node cluster can be monitored in real time. For example, a heartbeat mechanism can be used, where each node periodically sends a signal to the server to indicate that it is still alive and reports its status (e.g., "busy" or "idle"). If a node does not send a heartbeat for an extended period, it can be considered idle or offline. Resource sharing can also be implemented, with each node managing its own resource usage, such as CPU, memory, and storage. When the utilization of these resources falls below a set threshold, the node is considered idle. A load balancer can also be used to detect the node status. This allows for the determination of the number of nodes currently in an idle state within the node cluster.

[0121] In step S208, in response to the number of nodes in the idle state being greater than or equal to the number of pre-configured buckets, the server randomly assigns a target node to each data bucket from the nodes in the idle state.

[0122] Here, each data bucket is assigned to a different target node.

[0123] In this embodiment of the application, when the number of nodes in the idle state is greater than or equal to the number of pre-configured buckets, a target node is randomly assigned to each data bucket from the nodes in the idle state, and each data bucket is assigned to a different target node.

[0124] Steps 207 and 208 allow for real-time monitoring of the running status of each node in the node cluster to determine the number of idle nodes. When the number of idle nodes meets the pre-configured number of buckets, each data bucket is randomly assigned to a different idle node. Dynamically assigning a target node to each data bucket makes task allocation more reasonable, avoids resource waste and task backlog, improves resource utilization, and ensures efficient execution of computing tasks.

[0125] In step S209, the server processes multiple subtasks to be executed based on the task data in each data bucket through the target node.

[0126] In some embodiments, see Figure 8 , Figure 8 Step S209 can also be achieved through the following steps S2091 to S2094:

[0127] Step S2091: Determine the amount of computing resources required for the target node.

[0128] In this embodiment, computing resources refer to the total available computing power on the target node, which may include the following aspects: number of CPU cores, number of GPUs, memory size, storage capacity, and network bandwidth. The number of CPU cores represents the number of available central processing unit cores on the node; the more cores, the stronger the processing power. The number of GPUs represents the number of available graphics processing units on the node; GPUs are highly efficient at handling parallel computing tasks (such as deep learning). Memory size represents the available random access memory capacity on the node; the larger the memory, the more data can be processed simultaneously. Storage capacity represents the available storage space on the node (such as hard disk or solid-state drive capacity); the larger the storage capacity, the more data can be stored. Network bandwidth represents the node's network connectivity; the higher the bandwidth, the faster the data transmission speed. For example, node A has the following computing resources: 8 CPU cores, 2 GPUs, 32GB of memory, 1TB of storage capacity, and 1Gbps of network bandwidth.

[0129] Step S2092: Based on the amount of computing resources, determine the threshold number of subtasks to be executed by the target node in each round of task processing.

[0130] In this embodiment, the amount of computing resources required for each subtask to be executed can be determined. Based on the computing resources of the target node, a threshold number of subtasks that the target node can process in each round of task processing is calculated. For example, the computing resources of node A are: 8 CPU cores and 32GB of memory. The computing resources required for each subtask to be executed are 2 CPU cores and 4GB of memory. Therefore... 4*4=16<32, the threshold number of subtasks that the target node can process in each round of task processing is 4.

[0131] Step S2093: Obtain the priority of each subtask to be executed.

[0132] In this embodiment, the priority of each subtask to be executed is obtained. The priority of each subtask can be determined based on the execution logic of the task to be processed, or it can be preset; this embodiment does not limit this. For example, if there are 10 subtasks to be executed, their priorities can be preset according to the order in which they were generated. For instance, the first subtask generated has the highest priority.

[0133] Step S2094: Based on the quantity threshold and the task data in the data bucket allocated to the target node, perform at least one round of task processing on at least one subtask to be executed according to priority.

[0134] In this embodiment, during each round of task processing, subtasks to be executed that meet the target node's quantity threshold are assigned to the target node according to priority. Then, the task data in the data bucket assigned to the target node is used to process these subtasks. For example, if there are 10 subtasks to be executed, and the threshold for processing subtasks in each round is 4, then 4 subtasks are assigned to the target node according to priority, and these 4 subtasks are executed using the task data in the data bucket assigned to the target node. After processing, response data is obtained and saved. Then, the next round of task processing begins, and 4 subtasks are selected from the remaining 6 subtasks according to priority and assigned to the target node, and so on.

[0135] Through steps S2091 to S2094, the threshold number of subtasks that the target node can process in each round of task processing can be determined based on the computing resources of the target node. These subtasks are then allocated and processed according to priority, achieving efficient utilization of resources and orderly execution of tasks, thus ensuring the rationality of task allocation.

[0136] In some embodiments, after step 209, the following process may also be performed: in response to the absence of any subtasks to be executed in the target node, the target node is removed from the node cluster.

[0137] In this embodiment, after the server processes multiple subtasks to be executed based on the task data in each data bucket through the target node, the target node can be deleted from the node cluster when there are no more subtasks to be executed in the target node. For example, if the target node needs to process 10 subtasks to be executed, and after these 10 subtasks are processed, the target node has no other subtasks to be executed and can be deleted from the node cluster.

[0138] Step S210: The server obtains the data type of the response data obtained after task processing.

[0139] In this application embodiment, data types include unilateral data and multilateral data. The data type of the response data can be preset. Unilateral data refers to data generated by a task performed on a dataset held by a single data owner. For example, a bank possesses its customers' transaction records and credit score data, which belong entirely to the bank and are not shared with other institutions; when reading this data, it is unilateral data. Multilateral data refers to data generated by tasks performed on datasets held by multiple data owners. These datasets may overlap or be completely different in feature space or sample space. Federated learning, while protecting data privacy, shares model parameters or intermediate results securely (such as encryption, homomorphic encryption, etc.); these model parameters or intermediate results constitute multilateral data.

[0140] Step S211: The server determines the target receiver of the response data based on the data type.

[0141] Here, the task to be processed is a multi-terminal collaborative task, which is processed by multiple task processing terminals respectively; the target receiving terminal is at least one of the multiple task processing terminals.

[0142] In this embodiment, the task to be processed can be a multi-terminal collaborative task, which is processed by multiple task processing terminals respectively; the target receiving terminal is at least one of the multiple task processing terminals. When the data type is multi-party data, multiple task processing terminals can be determined according to the task to be processed, and the target receiving terminal can be one or more of the multiple task processing terminals. For example, suppose a medical research project is participated in by three hospitals (Hospital A, Hospital B, and Hospital C), and each hospital has its own patient data. This data cannot be directly shared, but through federated learning technology, each hospital performs tasks such as data preprocessing, feature extraction, model training, model evaluation, and model deployment locally, and shares model parameters through encryption technology. After Hospital A processes the task, it obtains response data, and the data type of the response data is determined to be multi-party data, thus identifying the target receiving terminals Hospital B and Hospital C.

[0143] In step S212, the server asynchronously transmits the response data to the target receiving end through a preset communication channel.

[0144] In this embodiment, a communication channel refers to a physical or logical path used for transmitting information in a communication system. The communication channel connects the sender and receiver, enabling the transmission of response data between them. Asynchronous transmission is a data communication method in which the sender and receiver do not need to be strictly synchronized in time, and the sending and receiving operations are independent of each other. For example, when using Pulsar to establish a trusted communication channel for asynchronous data transmission to the target receiver, the sender does not need to wait for confirmation from the target receiver after sending data; the message is buffered by the Pulsar middleware and reliably delivered.

[0145] Through steps 210 to 212, the target receiving end of the response data can be determined according to the data type, and the response data can be asynchronously transmitted to the target receiving end using a preset communication channel. This achieves efficient execution of multi-terminal collaborative tasks and secure data sharing. It ensures that each task processing end can obtain the required data in a timely manner in multi-terminal collaborative tasks, improving the efficiency and accuracy of task processing while protecting data privacy and security.

[0146] In this embodiment, firstly, the task dataset and pre-configured parameters for the task to be processed are obtained. Then, according to the number of pre-configured buckets in the pre-configured parameters, hash processing is used to evenly distribute the task data into multiple pre-configured data buckets, ensuring the uniformity of data distribution. The data after bucketing can be processed in parallel, improving data processing speed and thus increasing the processing speed of the task to be processed. Next, based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed. Then, the running status of each node in the node cluster is monitored in real time to determine the number of idle nodes. When the number of idle nodes meets the number of pre-configured buckets, each data bucket is randomly assigned to different idle nodes, dynamically assigning a target node to each data bucket, making task allocation more reasonable and avoiding resource waste and task backlog. Finally, based on the computing resources of the target node, the threshold number of sub-tasks to be executed that the target node can handle in each round of task processing is determined, and these sub-tasks are allocated and processed according to priority, realizing efficient use of resources and orderly execution of tasks, ensuring the rationality of task allocation. Subsequently, the target receiving end of the response data can be determined based on the data type, and the response data can be asynchronously transmitted to the target receiving end using a preset communication channel. This achieves efficient execution of multi-terminal collaborative tasks and secure data sharing. It ensures that each task processing end can obtain the required data in a timely manner during multi-terminal collaborative tasks, improving the efficiency and accuracy of task processing while protecting data privacy and security.

[0147] Figure 9 This is another optional flowchart illustrating the task processing method provided in the embodiments of this application, such as... Figure 9 As shown, the method includes the following steps S301 to S316:

[0148] Step S301: The terminal receives the task processing operation input by the user.

[0149] In step S302, the terminal encapsulates the task identifier into the task processing request.

[0150] In step S303, the terminal sends the task processing request to the server.

[0151] In step S304, the server responds to the task processing request by obtaining the task dataset and pre-configured parameters for the task to be processed.

[0152] Step S305: The server divides the task dataset of the task to be processed into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters.

[0153] Step S306: The server divides the task to be processed into multiple sub-tasks to be executed based on the pre-configured model parameters in the pre-configured parameters.

[0154] Step S307: The server determines the number of nodes in the node cluster that are currently in an idle state.

[0155] It should be noted that steps S301 to S307 are the same as steps S201 to S207 above. The implementation details of steps S301 to S307 in this embodiment will not be repeated.

[0156] In step S308, in response to the fact that the number of nodes in the idle state is less than the number of pre-configured buckets, the server allocates a target node for each data bucket from the nodes in the idle state.

[0157] Here, at least two data buckets are assigned to the same target node.

[0158] In this embodiment, when the number of idle nodes is less than the pre-configured number of buckets, a target node is randomly assigned to each data bucket from the idle nodes, and at least two data buckets are assigned to the same target node. For example, if the pre-configured number of buckets is 10, there are 10 data buckets. If there are only 5 idle nodes, two data buckets are assigned to each idle node.

[0159] Through steps S307 to 308, the running status of each node in the node cluster can be monitored in real time, and a target node can be dynamically allocated to each data bucket. When the number of idle nodes is less than the number of pre-configured data buckets, at least two data buckets are allowed to be allocated to the same idle node, which can effectively cope with resource shortages, ensure the flexibility and efficiency of task allocation, and optimize resource utilization.

[0160] Step S309: The server obtains the resource release amount of nodes in the node cluster that are in a non-idle state.

[0161] In this embodiment, the server can monitor the resource usage of each queried node in real time, and then, based on the computing resources of the target node, determine the resource release amount for nodes in the node cluster that are not idle. For example, if node A has 8 CPU cores and 32GB of memory, and its resource usage is 6 CPU cores and 12GB of memory, then the resource release amount for nodes in the node cluster that are not idle is determined to be 2 CPU cores and 20GB of memory.

[0162] In step S310, the server generates a new node based on the resource release amount if the resource release amount is greater than or equal to the amount of computing resources required for a subtask to be executed.

[0163] In this embodiment, when the resource release amount is greater than or equal to the computing resources required by a subtask to be executed, a new node can be generated based on the resource release amount. For example, the resource release amount is 2 CPU cores and 20GB of memory. The computing resources required by each subtask to be executed are 2 CPU cores and 4GB of memory. The number of CPU cores in the resource release amount is equal to the number of CPU cores required by the computing resources of a subtask to be executed, and the memory size in the resource release amount is greater than the memory size required by the computing resources of a subtask to be executed. Therefore, a new node can be generated based on the resource release amount.

[0164] Step S311: The server filters the target data bucket from at least two data buckets assigned to the same target node.

[0165] In this embodiment, when at least two data buckets are assigned to the same target node, a data bucket can be randomly selected. The target node executes the subtasks to be executed based on the task data in the current data bucket. After a new node is created, it is checked whether the subtasks to be executed in the current data bucket have been completed. If not, the current data bucket is used as the target data bucket. If completed, a data bucket is randomly selected from the remaining data buckets assigned to the same target node as the target data bucket. For example, there are two idle nodes and four data buckets. Data buckets 1 and 2 are assigned to node A, and node A executes the subtasks to be executed based on the task data of data bucket 1. Data buckets 3 and 4 are assigned to node B, and node B executes the subtasks to be executed based on the task data of data bucket 3. A new node C is added. It is checked whether the subtasks to be executed in data buckets 1 and 3 have been completed. If neither has been completed, a data bucket is randomly selected as the target data bucket. If one data bucket has not been completed, that data bucket is used as the target data bucket. If both have been completed, a data bucket is randomly selected from data buckets 2 and 4 as the target data bucket.

[0166] In step S312, the server assigns the target data bucket to the newly added node so that the newly added node can process at least one subtask to be executed in the task to be processed based on the task data in the target data bucket.

[0167] In this embodiment, after a target data bucket is assigned to a new node, the new node can process at least one subtask from the pending tasks based on the task data in the target data bucket. For example, there are two idle nodes and four data buckets. Data buckets 1 and 2 are assigned to node A, and node A executes the pending subtasks based on the task data of data bucket 1. Data buckets 3 and 4 are assigned to node B, and node B executes the pending subtasks based on the task data of data bucket 3. A new node C is added, and data bucket 1 is assigned to node C. There are a total of 10 pending subtasks. Assuming that node A has already executed 5 pending subtasks based on the task data of data bucket 1 and is currently executing the 6th pending subtask, node C can then execute the 7th pending subtask based on the task data of data bucket 1. It should be noted that when a node finishes executing all 10 pending subtasks based on the task data of data bucket 1, and an idle node appears, the data bucket with the most completed pending subtasks is preferentially assigned to the idle node.

[0168] Through steps S309 to S312, the resource usage of each node in the node cluster can be detected in real time, and the resource release amount of nodes in non-idle state can be calculated. When the resource release amount meets the computing resource requirements of the sub-task to be executed, a new node can be generated accordingly, and the target data bucket can be selected from multiple data buckets allocated to the same target node. The new node then executes the sub-task to be executed in the task to be processed. This not only improves the utilization rate of resources and ensures the efficient execution of tasks, but also enhances the flexibility and reliability of the system, making task processing more dynamic and efficient.

[0169] In step S313, the server processes multiple subtasks to be executed based on the task data in each data bucket through the target node.

[0170] Step S314: The server obtains the data type of the response data obtained after task processing.

[0171] Step S315: The server determines the target receiver of the response data based on the data type.

[0172] Here, the task to be processed is a multi-terminal collaborative task, which is processed by multiple task processing terminals respectively; the target receiving terminal is at least one of the multiple task processing terminals.

[0173] In step S316, the server asynchronously transmits the response data to the target receiving end through a preset communication channel.

[0174] It should be noted that steps S313 to S316 are the same as steps S209 to S212 above. The implementation details of steps S313 to S316 in this embodiment will not be repeated.

[0175] In this embodiment, firstly, the task dataset and pre-configured parameters for the task to be processed are obtained; then, according to the number of pre-configured buckets in the pre-configured parameters, hash processing is used to evenly distribute the task data into multiple pre-configured data buckets, which can ensure the uniformity of data distribution. The data after bucketing can be processed in parallel, improving the data processing speed and thus improving the processing speed of the task to be processed; then, based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed; and finally, by monitoring the running status of each node in the node cluster in real time, a target node is dynamically assigned to each data bucket. When the number of idle nodes is less than the pre-configured number of data buckets, at least two data buckets are allowed to be assigned to the same idle node. This effectively addresses resource constraints, ensuring flexibility and efficiency in task allocation while optimizing resource utilization. Furthermore, it allows real-time monitoring of resource usage across the node cluster, calculating the resource release amount for nodes in a non-idle state. When the resource release amount meets the computational resource requirements of the subtasks to be executed, a new node can be generated. The target data bucket is then selected from multiple data buckets allocated to the same target node, and the new node subsequently executes the subtasks within the pending tasks. This not only improves resource utilization and ensures efficient task execution but also enhances system flexibility and reliability, making task processing more dynamic and efficient. Finally, based on the target node's computational resources, a threshold number of subtasks that the target node can handle in each round of task processing is determined. These subtasks are then allocated and processed according to priority, achieving efficient resource utilization and orderly task execution, ensuring the rationality of task allocation. Subsequently, the target receiving end of the response data can be determined based on the data type, and the response data can be asynchronously transmitted to the target receiving end using a preset communication channel. This achieves efficient execution of multi-terminal collaborative tasks and secure data sharing. It ensures that each task processing end can obtain the required data in a timely manner during multi-terminal collaborative tasks, improving the efficiency and accuracy of task processing while protecting data privacy and security.

[0176] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0177] The federated scheduling systems used in the related technologies mainly include the following categories: Federated AI Technology Enabler (FATE), privacy computing frameworks, and PowerFL on Spark (an enhanced version of federated learning).

[0178] See Figure 10 , Figure 10This is an optional schematic diagram of the federated scheduling system provided in this application embodiment. The federated scheduling system scheme adopted by FATE is as follows: The federated intelligent technology enabling platform panel 1001 can be used for user interaction with the platform, and the federated intelligent technology enabling platform process 1002 is responsible for coordinating and managing the operation of the entire platform. The federated intelligent technology enabling platform process 1002 interacts with the database, operating system X, and federated machine learning module. The federated intelligent technology enabling platform process (Fate flow) is the core of FATE scheduling, scheduling federated jobs; FATE requires three parties, mainly divided into data application party (Guest), data provider (Host), and arbitrator (Arbiter); federated tasks are calculated and stored through the data engine; communication is conducted through operating system X (osx), supporting streaming or queued transmission.

[0179] See Figure 11 , Figure 11 This is another optional schematic diagram of the federated scheduling system provided in this application embodiment. The privacy computing framework is a privacy protection and machine learning framework. The federated scheduling system includes: a visual interface 1101, a privacy computing framework 1102, a reinforcement learning framework federation 1103, a reinforcement learning framework engine 1104, a cryptographic device 1105 (including a secure processing unit and a host equipment unit), and a plaintext device 1106. Ray can be used as the computing engine for computation and scheduling; a reinforcement learning framework federation (RayFed) can be used for hybrid compilation and scheduling, further encapsulating the Ray interface; multiple devices are supported, including plaintext computing units (PYU), secure processing units (SPU), and host equipment units (HEU). The SPU communicates based on a remote procedure call framework (brpc) and has a service mesh independent of the Ray network.

[0180] See Figure 12 , Figure 12This is another optional schematic diagram of the federated scheduling system provided in this application embodiment. PowerFL on Spark is a privacy-preserving and machine learning framework, and the federated scheduling scheme adopted is as follows: Spark is used as the computing engine, and distributed computing is performed through an elastic distributed dataset. Users submit tasks through a visual interface 1201. The streaming server 1202 receives the task and passes it to the Spark processing engine 1203. The Spark processing engine 1203 executes the task and sends the result to a message queue 1204. The message queue passes the message to another resource coordinator cluster or a container orchestration system cluster 1205. Asynchronous communication is performed through the Pulsar component; computing resources are managed by another resource coordinator (YARN) cluster as a resource manager, and the service layer relies on a container orchestration system (Kubernetes).

[0181] The FATE approach requires three roles (Guest, Host, and Arbiter). While Eggroll or Spark accelerates the efficiency of federated task storage and computation to some extent, FATE suffers from complex Domain Specific Language (DSL) syntax and network configuration, and lacks fine-grained control over task-level or actor-level dynamic scaling. Privacy-preserving computing frameworks can provide both encrypted and plaintext devices and initiate tasks via the Ray engine. However, aside from proxy communication, the SPU's reliance on brpc in privacy-preserving frameworks can lead to excessive port usage and resource waste during algorithm implementation. The PowerFL on Spark approach, similar to FATE, lacks fine-grained control over task-level or actor-level dynamic scaling, and its scheduling is less flexible than the Ray engine, limiting its support for deep learning.

[0182] The embodiments of this application address the aforementioned problems through the following aspects: First, by introducing the Ray engine, efficient scheduling of federated tasks is supported, making it suitable for handling large-scale parallel tasks and capable of supporting complex federated deep neural network (DNN) models. Furthermore, distributed optimization of algorithms can be achieved through simple, efficient, and flexible programming, ensuring distributed execution of algorithm runtime. Second, by utilizing the Ray engine's autoscaler component, Ray enables scaling operations across different platforms (such as Kubernetes) through provider interfaces. Combining this capability with Kubernetes' horizontal pod autoscaler (HPA) capability—that is, deeply integrating fine-grained control and autoscaler—further improves the utilization and efficiency of task computing resources, enabling fine-grained control over federated tasks. Finally, asynchronous communication is achieved by introducing the message middleware Pulsar. The Transport Layer Security (TLS) protocol ensures reliable communication channels between participants for cross-network transmission. Furthermore, only one port needs to be opened, and data streams and control task commands are transmitted asynchronously through Pulsar, achieving a highly unified abstraction and reducing development costs.

[0183] The following describes the task processing method of this application embodiment. This application embodiment designs a fine-grained automatic scaling federated scheduling system based on the Ray engine to support the federated modeling process among various data parties. First, the federated learning algorithm is distributed through the Ray engine. The data reading, writing, and computation tasks in federated learning can be run distributed across the head and worker nodes of the Ray cluster. The technologies involved include: a) distributing and parallelizing the reading of raw data stored on a distributed system (e.g., Hadoop Distributed File System, HDFS) through reinforcement learning framework data (Ray data); b) using Ray's remote function remote (@ray.remote, for example, @ray.remoteclass xx and @ray.remoteef xx) to delegate functions to other nodes for remote asynchronous execution. Through data hashing and bucketing, distributed transformation is performed, and each bucket (i.e., data bucket) can be executed in parallel. Then, by leveraging Ray's AutoScaler capabilities, fine-grained scaling of tasks is achieved, improving the efficiency and cost-effectiveness of computing resource provisioning. The technologies involved include: a) implementing Ray's Provider interface using Ray's AutoScaler abstraction, enabling dynamic scaling of Ray clusters across multiple platforms; b) achieving fine-grained control over Actors (stateful) and Tasks (stateless) in federated modeling tasks through Ray framework native support or functionality (Ray Native), scaling on Kubernetes clusters based on the actual number of resources required for each Actor and Task to run; c) using Ray's engine, combined with Ray Native commands, determining suitable nodes to be reclaimed based on the status of worker nodes in the cluster (down to whether a node is running a Task or Actor), thus saving resources. Finally, a communication channel is established through the message middleware Pulsar for information exchange among multiple parties. The technologies involved include: a) The data modeling party only needs to expose one public Pulsar port for other participants to connect to, and TLS encryption is used for establishing the communication channel; b) Task control instructions and algorithm operator instructions are uniformly managed and transmitted through Pulsar, simplifying the framework logic; c) Asynchronous communication and event-driven, that is, the way each participant obtains messages is asynchronous and does not require blocking, which can effectively improve the running efficiency.

[0184] See Figure 13 , Figure 13 This is another optional architecture diagram of the task processing system provided in the embodiments of this application. Figure 13 The system in the diagram can represent a federated scheduling system for any party participating in federated learning. First, a configuration file (e.g., YAML), or DSL file, is defined (the file defines the runtime of the algorithm to be run, data source, model parameters, etc.). The DSL file can include the task (i.e., the task to be processed) flow and the configuration of task parameters (i.e., pre-configured parameters). The DSL file is sent to gateway 1301 via a Hypertext Transfer Protocol (HTTP) request. Gateway 1301 then uses the federated scheduler 1302 to publish and schedule the federated task (i.e., the task to be processed). The federated task is a reinforcement learning framework task (Ray Job). The reinforcement learning framework cluster 1303 (Ray cluster, i.e., node cluster) is implemented using Kubernetes. For example, the YAML file has three independent modules: task reading, sample intersection, and model training. Although these three independent modules cannot be executed simultaneously, each module can be divided into multiple Tasks and Actors (here, Tasks and Actors are subtasks to be executed). The Tasks and Actors under each module can be executed in parallel. Each module has corresponding parameters (i.e., pre-configured model parameters) (defined by the user in the initial DSL file). The federated scheduler will parse these contents and submit them to the Ray cluster in the form of a task. The intermediate algorithm runtime is an algorithm image, and one module corresponds to one algorithm runtime.

[0185] The following is a detailed description of the task processing system described above. Ray data can be used for distributed reading and writing of the data source. Then, the read data is shuffled using a corresponding hash strategy (e.g., Murmur32) to divide it into multiple specified buckets (i.e., data buckets) (the number of buckets (i.e., the number of pre-configured buckets) can be specified in the initial DSL file). Nodes (i.e., target nodes) are assigned to each bucket for federated learning modeling or pre-processing (e.g., sample intersection). Kubernetes is used to calculate the remaining idle resources (i.e., resource release amount) in real time. If the remaining idle resources are greater than the resources required by the Task and Actor (i.e., the amount of computing resources required by a subtask to be executed), nodes can be added (i.e., new nodes are added). (For example, running a Task and an Actor requires 2GB of resources. If there are 10 buckets and only 3 nodes are allocated (i.e., nodes in an idle state), then those 3 nodes will first run all Tasks and Actors from those 3 buckets (Ray will try to balance the scheduling of these Tasks and Actors by default). If Kubernetes calculates that the remaining idle resources are greater than or equal to 2GB of resources, a fourth node can be added (i.e., a new node is added). When adding a node, there are two scenarios: First, if there are still Tasks and Actors in the queue waiting to be processed in the 3 buckets, the new node will prioritize executing those Tasks and Actors; second, if all Tasks and Actors in the 3 buckets can run on those 3 nodes (i.e., the resources on those 3 nodes are sufficient for the Tasks and Actors to run), then the new node will run the Tasks and Actors of the 4th bucket). The Actors and Tasks for each bucket are obtained based on the algorithm implementation. Each module (i.e., module task) (reading data, sample intersection, model training, and their corresponding parameter information) is obtained by parsing the YAML file and making a one-to-one correspondence.

[0186] For example, a typical longitudinal federated learning process can be divided into three stages: data retrieval, sample intersection, and model training. This process is identical for both parties involved in the federated learning (i.e., multiple task processing ends). During model training, the number of training rounds can be set, for example, 30 rounds, with each party training 30 times before the process ends. Each party then retains its own portion of the model. The input to model training is the output of sample intersection, and the input to sample intersection is the output of data retrieval. Therefore, the entire process is similar to a directed acyclic graph. During the computation within each module (a module may contain multiple Tasks or Actors, depending on the algorithm developer's specific settings), Tasks or Actors can be unilateral or bilateral. Unilateral tasks (e.g., retrieving data from a data source) do not involve communication, but bilateral tasks do. For example, if intermediate results need to be exchanged, a trusted communication channel needs to be established via Pulsar to transmit the data.

[0187] This application's embodiments can guide the development of a federated learning system framework. The main product form can be a federated learning platform providing services on public or private clouds, offering multiple business scenarios such as federated learning, advertising recommendation, and collaborative risk control, facilitating deep collaboration among various companies. This application's embodiments include the following key points:

[0188] First, the federated algorithm runtime is modified to be distributed and asynchronous. See [link / reference needed]. Figure 14 , Figure 14 This is a schematic diagram of an optional architecture for distributed asynchronous transformation provided in this application embodiment. The reinforcement learning framework cluster 1401 consists of a head node 1402 and multiple worker nodes 1403, and manages stateful objects and stateless tasks through distributed scheduling computation. Each node includes a scheduler, and each scheduler includes a scheduler program and object storage. First, distributed read and write operations of the data source are performed using Ray data, leveraging the flexible scheduling capabilities of the Ray engine compared to other engines. Then, the data is shuffled using an appropriate hashing strategy (e.g., Murmur32), divided into specified buckets, and Actors and Tasks are designed to achieve large-scale parallel processing of data.

[0189] Secondly, Ray Engine's AutoScaler component integrates with Kubernetes' HPA. Kubernetes HPA achieves automatic scaling by manipulating declaratively defined replicas, using Replica Sets (RS) to elastically scale containers (pods). However, it lacks fine-grained control similar to task or actor levels. Firstly, AutoScaler can automatically adjust the cluster size based on its load and provide fine-grained management by reclamating nodes running actors. Secondly, leveraging Ray Autoscaler's own Autoscaler abstraction requires implementing the Provider interface, enabling fine-grained scaling.

[0190] The pseudocode for fine-grained management can be defined as a class named NodeProvider, which includes: defining provider configuration and cluster name; retrieving the list of non-terminating nodes in the current cluster; checking whether a specified node is running; and cleaning up cluster resources. Fine-grained management can be achieved in this way.

[0191] Thirdly, a trusted communication channel is established through Pulsar, and control commands and data streams are uniformly transmitted via Pulsar. Firstly, in the federated modeling task, each algorithm component needs to communicate (encrypted gradient information, etc.). Then, since the federated task of the scheduling system in this embodiment may involve multi-party collaboration, task control commands are required, triggered asynchronously through the subscription mechanism provided by Pulsar. Finally, each worker node exchanges intermediate information about the algorithm's runtime according to a specified topic.

[0192] See Figure 15 , Figure 15This is a schematic diagram of an optional architecture for data transmission provided in an embodiment of this application. It involves two different services (Service A and Service B). Service A and Service B each have their own algorithm runtime for executing specific computational tasks. Each service has a federated scheduler responsible for managing communication and event triggering with the other service. Service A's federated scheduler subscribes to events from Service B and triggers events from Service A upon receiving an event. Service B's federated scheduler subscribes to events from Service A and triggers events from Service B upon receiving an event. Pulsar is a distributed messaging system used to transmit data between Service A and Service B. An event refers to an event channel used to transmit event messages. A topic refers to a data channel used to transmit data. Service A and Service B transmit data in batches via Pulsar. Data is encrypted via a TLS channel during transmission to ensure data transmission security.

[0193] The following description continues to illustrate the exemplary structure of the task processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the task processing device 455 of the memory 450 may include: a data acquisition module 4551, used to acquire the task dataset of the task to be processed and pre-configured parameters for the task to be processed; a data bucketing module 4552, used to divide the task dataset of the task to be processed into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters; each data bucket includes at least one task data of the task to be processed; a task partitioning module 4553, used to divide the task to be processed into multiple sub-tasks to be executed based on the pre-configured model parameters in the pre-configured parameters; a node allocation module 4554, used to allocate a target node to each data bucket from the node cluster; wherein each data bucket is allocated to one target node; and a task processing module 4555, used to perform task processing on the multiple sub-tasks to be executed through the target node based on the task data in each data bucket.

[0194] In some embodiments, the node allocation module 4554 is further configured to determine the number of nodes in the node cluster that are currently in an idle state; in response to the number of nodes in an idle state being greater than or equal to the number of pre-configured buckets, to randomly allocate a target node to each data bucket from the nodes in the idle state; wherein each data bucket is allocated to a different target node.

[0195] In some embodiments, the node allocation module 4554 is further configured to determine the number of nodes in the node cluster that are currently in an idle state; in response to the number of nodes in an idle state being less than the number of pre-configured buckets, to allocate a target node to each data bucket from the nodes in the idle state; wherein at least two data buckets are allocated to the same target node.

[0196] In some embodiments, the node allocation module 4554 is further configured to: obtain the resource release amount of nodes in the node cluster that are in a non-idle state; generate a new node based on the resource release amount in response to the resource release amount being greater than or equal to the amount of computing resources required for a subtask to be executed; filter a target data bucket from at least two data buckets allocated to the same target node; and allocate the target data bucket to the new node so that the new node can perform task processing on at least one subtask to be executed in the task to be processed based on the task data in the target data bucket.

[0197] In some embodiments, the task processing module 4555 is further configured to determine the amount of computing resources of the target node; based on the amount of computing resources, determine a threshold number of subtasks to be processed by the target node in each round of task processing; obtain the priority of each subtask to be processed; and based on the threshold number and the task data in the data bucket allocated to the target node, perform at least one round of task processing on the at least one subtask to be processed according to the priority.

[0198] In some embodiments, the device further includes a data transmission module, configured to acquire the data type of the response data obtained after task processing; determine the target receiving end of the response data based on the data type; wherein the task to be processed is a multi-terminal collaborative task, the multi-terminal collaborative task is processed by multiple task processing ends respectively; the target receiving end is at least one of the multiple task processing ends; and the response data is asynchronously transmitted to the target receiving end through a preset communication channel.

[0199] In some embodiments, the task partitioning module 4553 is further configured to partition the task to be processed into multiple module tasks based on the pre-configured model parameters in the pre-configured parameters; partition each module task into multiple sub-tasks; and determine each of the sub-tasks as the sub-task to be executed.

[0200] In some embodiments, the task partitioning module 4553 is further configured to partition the task to be processed into multiple module tasks based on the pre-configured model parameters in the pre-configured parameters; partition each module task into multiple sub-tasks; obtain the sub-tasks at the same task level in each module task to form a sub-task set; and determine all the sub-tasks in the sub-task set as a sub-task to be executed.

[0201] In some embodiments, the data bucketing module 4552 is further configured to perform hash processing on the task data in the task dataset to obtain the hash value of the task data; and allocate the task data in the task dataset to a pre-configured number of buckets according to the hash value of the task data to obtain multiple data buckets.

[0202] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the task processing method described in this application.

[0203] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the task processing method provided in this application. For example, ... Figure 3 The task processing method is shown.

[0204] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0205] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0206] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0207] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0208] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A task processing method, characterized in that, The method includes: Obtain the task dataset for the task to be processed and the pre-configured parameters for the task to be processed; Based on the number of pre-configured buckets in the pre-configured parameters, the task dataset of the task to be processed is divided into multiple data buckets; each data bucket includes at least one task data of the task to be processed. Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed; Assign a target node to each data bucket from the node cluster; whereby each data bucket is assigned to one target node. Through the target node, based on the task data in each data bucket, the multiple subtasks to be executed are processed.

2. The method according to claim 1, characterized in that, The allocation of target nodes for each data bucket in the slave node cluster includes: Determine the number of nodes in the node cluster that are currently in an idle state; In response to the number of nodes in the idle state being greater than or equal to the number of pre-configured buckets, a target node is randomly assigned to each data bucket from the nodes in the idle state; wherein each data bucket is assigned to a different target node.

3. The method according to claim 1, characterized in that, The allocation of target nodes for each data bucket in the slave node cluster includes: Determine the number of nodes in the node cluster that are currently in an idle state; In response to the number of nodes in the idle state being less than the number of pre-configured buckets, a target node is allocated to each data bucket from the nodes in the idle state; wherein at least two data buckets are allocated to the same target node.

4. The method according to claim 3, characterized in that, After allocating a target node for each data bucket from the nodes in the idle state, the method further includes: Obtain the resource release amount of the nodes in the node cluster that are in a non-idle state; In response to the resource release amount being greater than or equal to the computing resources required for a subtask to be executed, a new node is generated based on the resource release amount; Filter the target data bucket from at least two data buckets assigned to the same target node; The target data bucket is assigned to the newly added node so that the newly added node can perform task processing on at least one subtask of the task to be processed based on the task data in the target data bucket.

5. The method according to claim 1, characterized in that, The step of processing the multiple subtasks to be executed based on the task data in each data bucket through the target node includes: Determine the amount of computing resources required for the target node; Based on the amount of computing resources, determine the threshold number of subtasks to be executed that the target node can process in each round of task processing; Get the priority of each subtask to be executed; Based on the quantity threshold and the task data in the data bucket allocated to the target node, at least one round of task processing is performed on the at least one subtask to be executed according to the priority.

6. The method according to claim 5, characterized in that, After processing the at least one subtask to be executed in at least one round according to the priority based on the quantity threshold and the task data in the data bucket allocated to the target node, the method further includes: The data type of the response data obtained after the task processing is obtained; The target receiving end of the response data is determined based on the data type; wherein, the task to be processed is a multi-terminal collaborative task, which is processed by multiple task processing ends respectively; the target receiving end is at least one of the multiple task processing ends. The response data is asynchronously transmitted to the target receiving end through a preset communication channel.

7. The method according to any one of claims 1 to 6, characterized in that, Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed, including: Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple module tasks; Divide each module task into multiple sub-tasks; Each of the aforementioned subtasks is identified as the subtask to be executed.

8. The method according to any one of claims 1 to 6, characterized in that, Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple sub-tasks to be executed, including: Based on the pre-configured model parameters in the pre-configured parameters, the task to be processed is divided into multiple module tasks; Divide each module task into multiple sub-tasks; Obtain subtasks at the same task level from each module task and form a subtask set; All subtasks in the subtask set are identified as one subtask to be executed.

9. The method according to any one of claims 1 to 6, characterized in that, The task dataset of the task to be processed is divided into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters, including: The task data in the task dataset is hashed to obtain the hash value of the task data. Based on the hash value of the task data, the task data in the task dataset is allocated to a pre-configured number of buckets, resulting in multiple data buckets.

10. A task processing device, characterized in that, The device includes: The data acquisition module is used to acquire the task dataset of the task to be processed and the pre-configured parameters for the task to be processed; The data bucketing module is used to divide the task dataset of the task to be processed into multiple data buckets based on the number of pre-configured buckets in the pre-configured parameters; each data bucket includes at least one task data of the task to be processed. The task partitioning module is used to divide the task to be processed into multiple sub-tasks to be executed based on the pre-configured model parameters in the pre-configured parameters; The node allocation module is used to allocate a target node to each data bucket from the node cluster; wherein, each data bucket is allocated to one target node. The task processing module is used to process the multiple subtasks to be executed based on the task data in each data bucket through the target node.

11. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the task processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the task processing method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the task processing method according to any one of claims 1 to 9.

Images