A task processing method and apparatus

Abstract

The application discloses a task processing method and device, the method comprises the following steps: in response to a task processing instruction, a processing unit controls a first resource to perform a task processing operation through N virtual resource identifiers; wherein N comprises the number of data processing resources required by the task processing operation, the first resource comprises n data processing resources in a first state in a cluster; n is an integer greater than or equal to 1, and n is less than N; when a second resource is switched from a second state to the first state, the processing unit associates the second resource through N-n virtual resource identifiers; wherein the second processing resource comprises at least one data processing resource in the cluster except the first resource; the processing unit controls the first resource and the second resource to continue performing the task processing operation through the N virtual resource identifiers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a task processing method and device. BACKGROUND

[0002] In the distributed task processing process, only when the number of data processing resources in an idle state in a cluster is greater than or equal to the number of data processing resources required for task processing, can a task processing operation be performed. Such a task processing manner is not flexible enough. SUMMARY

[0003] In view of the above problems, embodiments of the present application provide a task processing method and device.

[0004] The technical scheme provided by the embodiments of the present application is as follows:

[0005] The embodiments of the present application provide a task processing method, which comprises the following steps:

[0006] In response to a task processing instruction, a processing unit controls a first resource to perform a task processing operation through N virtual resource identifiers; wherein N comprises the number of data processing resources required for the task processing operation, and the first resource comprises n data processing resources in a first state in a cluster; n is an integer greater than or equal to 1, and n is less than N;

[0007] In the case that a second resource is switched from a second state to the first state, the processing unit associates the second resource through N-n virtual resource identifiers; wherein the second processing resource comprises at least one data processing resource in the cluster except the first resource;

[0008] The processing unit controls the first resource and the second resource to continue to perform the task processing operation through the N virtual resource identifiers.

[0009] In some embodiments, the processing unit controls the first resource to perform the task processing operation through N virtual resource identifiers, comprising:

[0010] Obtaining association information; wherein the association information comprises mapping information between the N virtual resource identifiers and the data processing resources;

[0011] The processing unit associates n virtual resource identifiers to the first resource and associates N-n virtual resource identifiers to a third resource based on the association information; wherein the third resource comprises any data processing resource in the first resource;

[0012] The processing unit controls the first resource to perform the task processing operation through the N virtual resource identifiers in association with the association information.

[0013] In some embodiments, the processing unit controls the first resource to perform the task processing operation through the N virtual resource identifiers in association with the association information, including:

[0014] The processing unit obtains first data through the n virtual resource identifiers based on the association information; the first data includes processing result data of the first resource in the kth task processing process; the task processing operation includes multiple task processing processes; k is an integer greater than or equal to 1;

[0015] The processing unit obtains second data through the N-n virtual resource identifiers based on the association information; the second data includes preset data;

[0016] The processing unit processes the first data and the second data to obtain k+1 processing data.

[0017] In some embodiments, the processing unit controls the first resource to perform the task processing operation through the N virtual resource identifiers in association with the association information, further including:

[0018] The processing unit sends the k+1 processing data to the first resource through the n virtual resource identifiers based on the association information, so that the first resource performs the k+1th task processing process based on the k+1 processing data.

[0019] In some embodiments, the processing unit associates the second resource through N-n virtual resource identifiers, including:

[0020] The processing unit switches the association information between the target virtual resource identifier and the third resource to the association information between the target virtual resource identifier and the second resource, and associates the second resource through the association information between the target virtual resource identifier and the second resource; the target virtual resource identifier includes virtual resource identifiers in the N-n virtual resource identifiers that are the same as the number of data processing resources in the second resource.

[0021] In some embodiments, the processing unit controls the first resource and the second resource to continue to perform the task processing operation through the N virtual resource identifiers, including:

[0022] transmit third data obtained from the third resource to the second resource; wherein the third data comprises data corresponding to the third resource in the hth processing data obtained by the third resource from the processing unit; h is an integer greater than or equal to 1;

[0023] The processing unit controls the second resource to execute the hth task processing procedure based on the third data and the first resource to execute the hth task processing procedure based on the hth processing data through the n virtual resource identifiers.

[0024] In some embodiments, the obtaining the association information comprises:

[0025] obtaining first address information and second address information; wherein the first address information comprises address information of the data processing resource; and the second address information comprises the N virtual resource identifiers.

[0026] The processing unit obtains the association information based on the first address information and the second address information.

[0027] In some embodiments, the method further comprises:

[0028] The processing unit creates N virtual resources based on the task processing instruction and sets virtual resource identifiers for the N virtual resources to obtain the N virtual resource identifiers.

[0029] In some embodiments, the method further comprises:

[0030] After the processing unit controls the first resource to execute the task processing operation through the N virtual resource identifiers, the state of the first resource is adjusted to the second state; and after the processing unit controls the first resource and the second resource to continue to execute the task processing operation through the N virtual resource identifiers, the state of the second resource is adjusted to the second state.

[0031] Embodiments of the present application also provide a task processing device, comprising:

[0032] A processing unit is configured to control a first resource to execute a corresponding task processing operation through N virtual resource identifiers in response to a task processing instruction; wherein N comprises a number of data processing resources required by the task processing operation, and the first resource comprises n data processing resources in a cluster in a first state; n is an integer greater than or equal to 1, and n is less than N.

[0033] The processing unit is further configured to associate the second resource with the N-n virtual resource identifiers in a case that the second resource is switched from the second state to the first state, wherein the second processing resource comprises at least one data processing resource in the cluster except the first resource.

[0034] The processing unit is further configured to control the first resource and the second resource to continue the task processing operation by the N virtual resource identifiers.

[0035] The application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor of an electronic device, and can implement the task processing method according to any one of the preceding embodiments.

[0036] The task processing method provided by the application can still execute the task processing operation when the number of data processing resources in the first state in the cluster is less than the number of data processing resources required by the task processing operation, and can dynamically and flexibly associate the available data processing resources to the task processing operation to continue the task processing operation when at least one data processing resource in the cluster is switched to the first state. Therefore, the task processing operation can be flexibly executed when the number of available data processing resources cannot meet the requirement of the task processing operation, and the data processing resources can be dynamically added during the task processing operation. Thus, the flexibility of the task processing operation based on the cluster is greatly improved, and the efficiency of the task processing operation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The flowchart of the task processing method provided by the application is shown.

[0038] Figure 2 The flowchart of the processing unit provided by the application for associating and controlling the first resource to execute the task processing operation by the N virtual resource identifiers is shown.

[0039] Figure 3 The flowchart of the processing unit provided by the application for associating and controlling the first resource to execute the task processing operation by the N virtual resource identifiers based on the association information is shown.

[0040] Figure 4 The structural diagram of the task processing method provided by the application is shown.

[0041] Figure 5 The structural diagram of the task processing device provided by the application is shown. DETAILED DESCRIPTION

[0042] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below.

[0043] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0044] The distributed task processing scheme with the aid of cluster data processing resources has been widely applied, such as in the model distributed training scheme of the cluster, each sub-task of the distributed training is independently completed on the device of the cluster, and the training gradient is combined after the training task of each device is completed, which can greatly improve the efficiency of model training.

[0045] However, in actual application, only when the number of idle devices or resources in the cluster is greater than or equal to the number of devices or resources required by the distributed task processing process, the distributed task processing process can be started, otherwise, if the number of idle devices or resources in the cluster is less than the number of devices or resources required by the distributed task processing process, the distributed task processing process cannot be started, and can only continue to wait for idle devices or resources until the number of idle devices or resources can meet the needs of the distributed task processing process.

[0046] In order to solve the above technical problems, the technical scheme of setting the distributed task as a pre-emptible task is provided in the related art, that is, when the current distributed task is in a waiting state due to insufficient idle devices or resources in the cluster, if the number of idle devices or resources in the cluster is greater than or equal to the number of devices or resources required by other tasks after the current distributed task in the task queue, the execution of other distributed tasks can be started, and until the number of idle devices or resources in the cluster can meet the needs of the current distributed task, the execution of other distributed tasks is paused and the execution of the current distributed task is started.

[0047] However, in the execution process of the above scheme, if the number of devices or resources required by the current distributed task is large, the waiting time of the current distributed task may be long, and the specific waiting time cannot be determined; and in the process of starting and pausing other distributed tasks, the data processing resources of the cluster are also easily consumed, and at the same time, in the execution process of the above scheme, the execution of other distributed tasks must be in a pre-emptible mode. Therefore, the above scheme cannot fundamentally solve the technical problem of flexible execution of the current distributed task and other distributed tasks.

[0048] Based on the above problems, the embodiment of the present application provides a task processing method and device. The task processing method provided by the embodiment of the present application can execute the task processing operation based on the limited available data processing resources when the available data processing resources in the cluster cannot completely meet the task processing demand, and can dynamically, flexibly and timely associate the data processing resources that are switched to the available state to the task processing operation process when it is detected that any data processing resource in the cluster is switched to the available state, thereby realizing the dynamic and flexible processing of the task processing operation.

[0049] It should be noted that the task processing method described above can be implemented by a processor of an electronic device. For example, the electronic device can include a device for starting, managing and controlling the task processing operation. For example, the processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor.

[0050] Figure 1 The flowchart of the task processing method provided by the embodiment of the present application is shown in FIG. 1, which can include steps 101 to 103. Figure 1

[0051] In step 101, in response to a task processing instruction, a processing unit controls a first resource to execute a task processing operation through N virtual resource identifiers.

[0052] In the step, N includes the number of data processing resources required by the task processing operation; the first resource includes n data processing resources in a first state in the cluster; n is an integer greater than or equal to 1; and n is less than N.

[0053] In an embodiment, the first state can include a state in which at least part of the resource is available. For example, the first state can include an idle state.

[0054] In an embodiment, the task processing instruction can be sent by a host of the processing unit. For example, the task processing instruction can be input to the processing unit by a user or a professional technician. ​

[0055] In an embodiment, the processing unit can include an electronic device that is communicatively connected with the cluster; for example, the processing unit can be a physical machine device or a virtual machine device; for example, the processing unit can implement a unit that controls the start, execution, pause, restart, and stop of the task processing operation; for example, the processing unit can include a parameter server (PS) in a PS-Worker architecture.

[0056] In an embodiment, the N virtual resource identifiers can include identifiers of N independent virtual resources; for example, the virtual resource identifier can be set by the processing unit, such as a virtual Internet Protocol (IP) address and / or a port number.

[0057] In an embodiment, the data processing resource can include a physical and / or virtual simulation resource that can implement a data processing function; for example, when the number of data processing resources is multiple, the types of the data processing resources can be different or the same, such as a first data processing resource can be a CPU, and a second data processing resource can be a Graphics Processing Unit (GPU); for example, a third data processing resource can be a physical resource, and a fourth data processing resource can be a virtual simulation resource; or for example, multiple data processing resources can be multiple GPUs.

[0058] In an embodiment, the cluster can include multiple data processing resources; for example, in a PS-Worker architecture, the cluster can be a worker server of the PS; for example, the cluster can include any cluster that can implement a distributed data processing function, such as a Kubernetes cluster.

[0059] In an embodiment, a communication connection can be established between the cluster and the processing unit, and after receiving the task processing instruction, the processing unit can issue the task processing instruction to the cluster through the communication connection, and the cluster can distribute the task processing operation corresponding to the task processing instruction to at least one data processing resource, so that the at least one data processing resource executes the task processing operation and returns the task processing result.

[0060] In an embodiment, the task processing instruction can include the number of data processing resources required by the task processing operation, and can also include the type of data processing resource required by the task processing operation; for example, the task processing instruction can also include information such as the timing of starting the task processing operation, which is not limited in the embodiments of the present application.

[0061] Exemplarily, the processing unit can associate the first resource through the N virtual resource identifiers to control the first resource to perform the task processing operation, which can be implemented in any of the following ways:

[0062] The processing unit associates the first resource through the N virtual resource identifiers, and after the association succeeds, controls at least part of the resources in the first resource to perform the task processing operation through the N virtual resource identifiers.

[0063] The processing unit associates the first resource through the N virtual resource identifiers, and simultaneously analyzes the task processing operation to determine a target sub-task, and then controls the first resource to perform the target sub-task through the N virtual resource identifiers.

[0064] In step 102, in the case that the second resource is switched from the second state to the first state, the processing unit associates the second resource through the N-n virtual resource identifiers.

[0065] The second resource includes at least one data processing resource in the cluster except the first resource.

[0066] Exemplarily, if no data processing resource in the cluster is switched from the second state to the first state, the processing unit can not perform the operation of associating the second resource through the N-n virtual resource identifiers.

[0067] In an embodiment, the second state can include a state in which at least part of the data processing resources are unavailable; exemplarily, the second state can include a data processing state; exemplarily, the second state can include a state occupied by other task processing operations.

[0068] In an embodiment, the processing unit or the cluster can monitor the states of the data processing resources in the cluster, and in the case that at least one data processing resource is detected to be switched to the first state, determine the at least one data processing resource as the second resource.

[0069] Exemplarily, the processing unit can associate the second resource through the N-n virtual resource identifiers in any of the following ways:

[0070] In the case that the number of resources is less than N-n, the processing unit acquires the number of resources, and associates the virtual resource identifier of the number of resources in the N-n virtual resource identifiers to the second resource; wherein the number of resources can include the number of data processing resources in the second resource.

[0071] In the case that the number of resources is equal to N-n, the processing unit associates the N-n virtual resource identifiers with the data processing resources in the second resource.

[0072] When the number of resources is greater than Nn, the processing unit associates the Nn virtual resource identifiers with the corresponding Nn data processing resources in the second resource.

[0073] Step 103: The processing unit controls the first resource and the second resource to continue performing task processing operations through N virtual resource identifiers.

[0074] In one implementation, the processing unit may add a second resource to the data processing resource set of the task processing operation.

[0075] In one implementation, the processing unit can analyze the processing status of the task processing operation to obtain multiple sub-tasks, and then assign each data processing resource to be processed through N virtual resource identifiers as the first resource and the second resource, thereby continuing to execute the task processing operation.

[0076] As can be seen from the above, the task processing method provided in this application embodiment, in response to the task processing instruction, under the condition that the number of data processing resources in the first state in the cluster is less than the number of data processing resources required for the task processing operation, the processing unit associates and controls the n data processing resources in the first state in the cluster to perform the task processing operation through N virtual resource identifiers, and when the second resource switches from the second state to the first state, the processing unit associates the second resource through Nn virtual resource identifiers, and controls the first resource and the second resource to continue to perform the task processing operation through N virtual resource identifiers.

[0077] Therefore, the task processing method provided in this application embodiment can still perform task processing operations even when the number of data processing resources in the first state of the cluster is less than the number of data processing resources required for the task processing operation. Furthermore, when at least one data processing resource in the cluster switches to the first state, the available data processing resources can be dynamically and flexibly associated with the task processing operation to continue executing the task processing operation. This achieves flexible execution of task processing operations when the number of available data processing resources cannot meet the requirements of the task processing operation, as well as dynamic addition of data processing resources during the task processing operation, thereby greatly improving the flexibility and efficiency of task processing operations on the cluster basis.

[0078] The task processing method provided in this application embodiment may further include the following operations:

[0079] The processing unit creates N virtual resources based on the task processing instructions and sets virtual resource identifiers for the N virtual resources, thus obtaining N virtual resource identifiers.

[0080] In an embodiment, after receiving the task processing instruction, the processing unit can analyze the task processing instruction, obtain the number of data processing resources required by the task processing operation, and determine the number of virtual resources to be created based on the number of data processing resources required by the task processing operation.

[0081] In an embodiment, after the creation of the N virtual resources is completed, the processing unit can set a virtual resource identifier for each virtual resource, thereby obtaining N virtual resource identifiers.

[0082] In an embodiment, the virtual resource can be a virtual object of the data processing resource required by the task processing operation created by the processing unit; for example, the virtual resource can be an agent object of the data processing resource on the side of the processing unit.

[0083] As can be seen from the above, in the task processing method provided by the embodiments of the present application, the processing unit can create N virtual resources according to the task processing instruction, and set a virtual resource identifier for each virtual resource, thereby obtaining N virtual resource identifiers. Thus, in the task processing method provided by the embodiments of the present application, the processing unit can flexibly determine the number of virtual resources to be created according to the task processing instruction received by the processing unit, thereby laying a foundation for the subsequent processing unit to associate and control the data processing resource to execute the task processing operation through the virtual resource identifier.

[0084] Based on the foregoing embodiments, the task processing method provided by the embodiments of the present application can further include the following operations:

[0085] After the processing unit associates and controls the first resource to execute the task processing operation through the N virtual resource identifiers, the state of the first resource is adjusted to a second state; and after the processing unit controls the first resource and the second resource to continue to execute the task processing operation through the N virtual resource identifiers, the state of the second resource is adjusted to the second state.

[0086] For example, the second state can include a state in which the data processing resource is occupied by the task processing operation; for example, the second state can include a state in which the data processing resource cannot be occupied by other task processing operations.

[0087] As can be seen from the above, in the task processing method provided by the embodiments of the present application, after the processing unit associates and controls the first resource and the second resource to execute the task processing operation, the state of the first resource and the second resource can be adjusted to the second state in real time, thereby reducing the risk of the first resource and the second resource being requested or occupied by other task processing operations, reducing the probability of conflict or mutual exclusion of different task processing operations in terms of data processing resources, and thereby improving the stability and efficiency of the processing of the task processing operation.

[0088] Based on the foregoing embodiments, the task processing method provided by the embodiments of the present application can be used to control the first resource to perform the task processing operation by the N virtual resource identifiers in association with the first resource, and the first resource can be controlled to perform the task processing operation by the N virtual resource identifiers in association with the first resource through the following steps Figure 2 The flow shown in FIG. 8 can include steps 801 to 803. Figure 2 The flow shown in FIG. 8 can include steps 801 to 803. Figure 2 The flow shown in FIG. 8 can include steps 801 to 803.

[0089] Step 201: Obtain association information.

[0090] The association information includes mapping information between the N virtual resource identifiers and the data processing resources.

[0091] In an embodiment, the association information can be stored in the storage space of the processing unit. For example, the association information can be obtained by the processing unit from other devices or other storage devices.

[0092] In an embodiment, the association information can include one-to-one mapping information between the N virtual resource identifiers and the N data processing resources. For example, the association information can include a many-to-one mapping relationship between the N virtual resource identifiers and M data processing resources, where M is a positive integer less than N.

[0093] Step 202: The processing unit associates the n virtual resource identifiers with the first resource and the N-n virtual resource identifiers with the third resource based on the association information.

[0094] The third resource includes any data processing resource in the first resource.

[0095] In an embodiment, the third resource can include one data processing resource in the first resource. In this case, the association information can include one-to-one mapping information between the n virtual resource identifiers and the data processing resource in the first resource, and can also include a many-to-one mapping relationship between the N-n virtual resource identifiers and the third resource.

[0096] In an embodiment, when n is greater than or equal to 2, the third resource can include at least two data processing resources in the first resource. In this case, the association information can include one-to-one mapping information between the n virtual resource identifiers and the data processing resources in the first resource, and can also include a many-to-one or one-to-one mapping relationship between the N-n virtual resource identifiers and the third resource.

[0097] Step 203: The processing unit controls the first resource to perform the task processing operation by the N virtual resource identifiers in association with the first resource based on the association information.

[0098] Exemplarily, when the processing unit needs to start or execute a task processing operation, the processing unit can obtain the mapping relationship between the N virtual resource identifiers and the respective data processing resources in the first resource from the association information, and based on the mapping relationship, allocate sub-tasks to the respective data processing resources in the first resource through the N virtual resource identifiers, so as to execute the task processing operation.

[0099] As can be seen from the above, in the task processing method provided by the embodiments of the present application, after the processing unit obtains the mapping information (i.e. the association information) between the N virtual resource identifiers and the data processing resources, the processing unit can associate the N virtual resource identifiers to the first resource and associate the N-n virtual resource identifiers to the third resource based on the association information, and then control the first resource to execute the task processing operation through the N virtual resource identifiers based on the association information.

[0100] Therefore, in the task processing method provided by the embodiments of the present application, the processing unit can determine how to associate the virtual resource identifiers with the data processing resources based on the association information, and can also control the first resource to execute the task processing operation through the virtual resource identifiers, so as to not only realize the isolation between the task processing operation and the processing unit, reduce the coupling degree between the task processing operation and the processing unit, but also improve the flexibility of the processing unit in controlling the task processing operation.

[0101] Based on the foregoing embodiments, in the task processing method provided by the embodiments of the present application, the association information can be obtained by implementing steps A1 to A2:

[0102] Step A1, obtain first address information and second address information.

[0103] The first address information includes address information of the data processing resources; and the second address information includes the N virtual resource identifiers.

[0104] In an implementation manner, the virtual resource identifier can include address information of a virtual resource.

[0105] In an implementation manner, the first address information can include physical address information of the data processing resources; exemplarily, the first address information can include network address information of the data processing resources, such as an IP address of the data processing resources, and the first address information can also include a port number of the data processing resources.

[0106] In an implementation manner, the processing unit can obtain the first address information from the cluster; exemplarily, the cluster can actively send the first address information to the processing unit; exemplarily, after receiving a task processing instruction and determining the number of data processing resources required for the task processing operation, the processing unit can send a request message for obtaining the first address information to the cluster, and after receiving the request message, the cluster can send the first address information to the processing unit.

[0107] In an embodiment, after the processing unit creates the N virtual resources, the processing unit can set IP addresses and port numbers for the N virtual resources, and establish a mapping relationship between the IP addresses and port numbers and the first address information.

[0108] Step A2, the processing unit obtains the association information based on the first address information and the second address information.

[0109] In an embodiment, the processing unit can determine how to establish the association information between the first address information and the second address information based on the number of addresses in the first address information and the second address information; for example, if the number of the first address information is less than the number of the second address information, the association information can include a one-to-one mapping relationship between the first address information and the second address information, and a one-to-many mapping relationship between the first address information and the second address information, such as the number of data processing resources in the first resource is less than the number of virtual resources, at this time, the association information can include a one-to-one mapping relationship between n data processing resources and n virtual resource identifiers, and a one-to-many mapping relationship between the data processing resources and N-n virtual resource identifiers.

[0110] In an embodiment, the virtual resource identifier can include the IP address and the port number of the virtual resource, in which case the virtual resource identifier can not only uniquely represent the virtual resource, but also represent the address of the virtual resource, so that the association information can reflect the mapping relationship between the address of the virtual resource and the address of the data processing resource, thereby improving the efficiency of the processing unit in data interaction between the virtual resource identifier and the data processing resource.

[0111] As can be seen from the above, the task processing method provided by the embodiments of the present application can obtain the association information between the address information of the data processing resource and the N virtual resource identifiers after obtaining the address information of the data processing resource and the N virtual resource identifiers.

[0112] Therefore, the task processing method provided by the embodiments of the present application improves the flexibility of the association information, greatly improves the flexibility of the processing unit in associating the corresponding data processing resource for different task processing operations, and enables the processing unit to determine the association information based on the address information of the data processing resource and the address information of the virtual resource, so that the association information is not only related to the actual task processing operation, but also related to the state and number of the data processing resource in the cluster, thereby improving the stability of the processing unit in executing the task processing operation based on the association information through the virtual resource identifier.

[0113] Based on the foregoing embodiments, the task processing method provided in the embodiments of the present application can control the first resource to perform the task processing operation through the N virtual resource identifiers based on the association information, and the first resource can be controlled to perform the task processing operation through the N virtual resource identifiers based on the association information Figure 3 The flow shown in FIG. 3 can include steps 301 to 303. Figure 3 The flow shown in FIG. 3 can include steps 301 to 303. Figure 3 The flow shown in FIG. 3 can include steps 301 to 303.

[0114] In step 301, the processing unit obtains first data through the n virtual resource identifiers based on the association information.

[0115] The first data includes the processing result data of the first resource in the kth task processing process; the task processing operation includes a plurality of task processing processes; and k is an integer greater than or equal to 1.

[0116] In an embodiment, the first data can include data obtained by the first resource by processing data received from the processing unit in the kth task processing process. For example, the first data can include gradient data calculated by each data processing resource in the first resource in the kth task processing process.

[0117] In an embodiment, the task processing operation that needs to be performed by the cluster can include a plurality of task processing processes, and the data to be processed in each task processing process can be obtained from the processing unit. For example, in a distributed training task using a PS-Worker architecture, the processing unit can be a PS, and the Worker can include a plurality of data processing resources in the cluster. At this time, the Worker can pull weight data from the PS and process the weight data to obtain gradient data.

[0118] In an embodiment, the processing unit can obtain the processing result data of each data processing resource from the first resource based on the association information.

[0119] In step 302, the processing unit obtains second data through the N-n virtual resource identifiers based on the association information.

[0120] The second data includes preset data.

[0121] For example, since the number n of data processing resources in the first resource is less than N, the n data processing resources perform the kth task processing operation during the execution of the task processing operation, and the N-n virtual resource identifiers do not correspond to actually available data processing resources. Therefore, the virtual resource corresponding to the N-n virtual resource identifier can send preset data to the target address, thereby improving the stability of the task processing operation.

[0122] In an embodiment, the preset data can be empty gradient data; for example, the preset data can be adjusted according to the actual task processing operation, and the embodiment of the present application is not limited thereto.

[0123] For example, the order of steps 301 and 302 can be adjusted, or they can be executed in parallel, and the embodiment of the present application is not limited thereto.

[0124] Step 303, the processing unit processes the first data and the second data to obtain the k+1 processing data.

[0125] In an embodiment, the processing unit can process the first data and the second data based on the task processing requirement of the task processing operation; for example, the task processing requirement can include at least one of an algorithm for processing the first data and the second data, an expected time of the task processing operation, a convergence requirement of the task processing operation, and a number and type of parameters associated with the task processing operation, and the embodiment of the present application is not limited thereto.

[0126] As can be seen from the above, in the task processing method provided by the embodiment of the present application, the processing unit obtains the first data based on the associated information through n virtual resource identifiers, and then obtains the second data based on the associated information through N-n virtual resource identifiers, and processes the first data and the second data to obtain the k+1 processing data.

[0127] Therefore, in the task processing method provided by the embodiment of the present application, the processing unit can not only obtain the data processing result of the first resource through N virtual resource identifiers, but also obtain the preset data, that is, the processing unit executes the business processing operation by means of a small amount of available data processing resources in the cluster through N virtual resource identifiers, and in combination with the preset data obtained through N-n virtual resource identifiers, the stability of the processing unit in managing the business processing operation can also be improved.

[0128] Based on the foregoing embodiment, in the task processing method provided by the embodiment of the present application, the processing resource controls the first resource to execute the task processing operation based on the associated information through N virtual resource identifiers, which can further include the following operations:

[0129] The processing unit sends the k+1 processing data to the first resource based on the associated information through n virtual resource identifiers, so that the first resource executes the k+1 task processing process based on the k+1 data.

[0130] For example, the k+1 processing data can include weight data to be processed in the k+1 task processing process obtained by processing the first data and the second data.

[0131] Exemplarily, the processing unit can send the (k+1)th processing data to the data processing resource in the cluster through the N virtual resource identifiers based on the association information, and since the actually available data processing resource is the first resource, the data processing resource actually receiving the (k+1)th data is the first resource.

[0132] As can be seen from the above, in the task processing method provided by the embodiments of the present application, the processing unit sends the (k+1)th processing data to the first resource through the N virtual resource identifiers based on the association information, so that the first resource performs the (k+1)th task processing process based on the (k+1)th processing data. Thus, the task processing method provided by the embodiments of the present application realizes the multiple task processing processes in the task processing operation through the small number of available data processing resources in the cluster, and realizes the smooth closed-loop processing between the processing unit and the small number of available data processing resources.

[0133] Based on the foregoing embodiments, in the task processing method provided by the embodiments of the present application, the processing unit associates the second resource through the N-n virtual resource identifiers, which can be realized in the following ways:

[0134] The processing unit switches the association information between the target virtual resource identifier and the third resource to the association information between the target virtual resource identifier and the second resource, and associates the second resource through the association information between the target virtual resource identifier and the second resource.

[0135] The target virtual resource identifier includes the virtual resource identifiers in the N-n virtual resource identifiers which are the same as the number of data processing resources in the second resource.

[0136] Exemplarily, the number p of data processing resources in the second resource can be an integer greater than or equal to 1 and less than or equal to N-n.

[0137] In an implementation, when p is less than N-n, p target virtual resource identifiers corresponding to the N-n virtual resource identifiers can be determined, and then the association between the p virtual resource identifiers and the third resource is switched to the association between the p virtual resource identifiers and the second resource; exemplarily, the association information corresponding to the N-n-p virtual resource identifiers can be kept unchanged.

[0138] In an implementation, when p is equal to N-n, that is, the number of data processing resources included in the second resource is N-n, the number of target virtual resource identifiers is the same as the number of data processing resources in the second resource, and the association information between the target virtual resource identifiers and the second resource after switching can include one-to-one mapping information between the target virtual resource identifiers and the second resource.

[0139] From the above, in the task processing method provided by the embodiment of the application, after the second resource is detected, the processing unit can associate the target virtual resource identifier with the second resource in real time, switch to the association between the target virtual resource and the second resource, and associate the second resource through the association between the target virtual resource identifier and the second resource.

[0140] Based on the foregoing embodiment, in the task processing method provided by the embodiment of the application, the processing unit controls the first resource and the second resource to continue the task processing operation through the N virtual resource identifiers, which can be implemented through steps B1 to B2:

[0141] Step B1: transmitting third data obtained from the third resource to the second resource.

[0142] The third data includes data corresponding to the third resource in the hth processing data obtained by the third resource from the processing unit; and h is an integer greater than or equal to 1.

[0143] In an implementation manner, the hth processing data output by the processing unit can include a weight parameter required to be processed by a data processing resource in the first resource in the hth data processing process.

[0144] In an implementation manner, the weight parameters required to be processed by the data processing resources in the first resource in the kth data processing process can be different, that is, the hth processing data can include a plurality of data components, and each data component can correspond to a different data processing resource in the first resource.

[0145] In an implementation manner, since the target virtual resource identifier is associated with the third resource before being associated with the second resource, the third data obtained by the third resource can be transmitted to the second resource, so as to realize sharing of the to-be-processed data of the third resource and the second resource in the hth data processing process, improve the stability of the hth data processing process, and enable the second resource to participate in the task processing process of the task processing operation immediately after switching to the second state, thereby improving the execution efficiency of the task processing operation.

[0146] Step B2: the processing unit controls the second resource to perform the hth task processing process based on the third data through the target virtual resource identifier, and controls the first resource to perform the hth task processing process based on the hth processing data through the N virtual resource identifiers.

[0147] In an implementation manner, the processing unit can control the second resource to start the task processing operation through the target virtual resource identifier, and perform the hth task processing process based on the third data, and simultaneously control the first resource to perform the hth task processing process based on the hth processing data through the N virtual resource identifiers.

[0148] Exemplarily, after the first resource and the second resource execute the hth task processing process, the hth processing result can be obtained, and the hth processing result is returned to the processing unit through the target virtual resource identifier and the n virtual resource identifiers, so that the processing unit obtains the (h+1)th processing data based on the hth processing result, and the processing unit sends the (h+1)th processing data to the second resource and the first resource through the target virtual resource identifier and the n virtual resource identifiers, so as to realize the closed loop execution of the task processing operation between the processing unit and the first resource and the second resource.

[0149] As can be seen from the above, in the task processing method provided by the embodiment of the application, after the second resource is switched to the second state and the second resource is associated with the target virtual resource identifier, under the condition that the second resource obtains the third data from the third resource, the processing unit can control the second resource through the target virtual source identifier and control the first resource through the n virtual resource identifiers to execute the hth task processing process, so as to realize that the second resource is dynamically added to the task processing process of the task processing operation without interrupting or suspending the task processing process of the first resource in the task processing process, and realize the dynamic, flexible and smooth addition of the data processing resource in the execution process of the task processing operation, thereby greatly improving the execution efficiency of the task processing operation.

[0150] Figure 4 The structure diagram realized by the task processing method provided by the embodiment of the application is shown in Figure 4 The processing unit 401 can be a PS in the PS-Worker, and the cluster 402 can include a plurality of Wokers in the PS-Worker architecture; exemplarily, the cluster 402 can include a plurality of data processing resources; exemplarily, the data processing resources in the cluster 402 can include GPUs, and in Figure 4 the cluster 402 can include a plurality of GPUs.

[0151] Exemplarily, when the processing unit 401 receives the task processing instruction, the GPUs in the first state in the cluster 402 can include the first GPU 4021 and the second GPU 4022.

[0152] Exemplarily, the processing unit 401 can determine that the number of data processing resources required by the task processing operation corresponding to the task processing instruction is 4 after analyzing the task processing instruction, at this time, the processing unit 401 can create a virtual resource 403 based on the number of data processing resources required by the task processing operation, wherein the virtual resource 403 can include a first virtual resource 4031, a second virtual resource 4032, a third virtual resource 4033 and a fourth virtual resource 4034, and the first virtual resource 4031 to the fourth virtual resource 4034 are set with virtual resource identifiers, that is, address information of the virtual resource, that is, the second address information.

[0153] For example, the number of GPUs in the cluster 402 in the first state is less than the number of GPUs required for the task processing operation, and therefore the processing unit 401 can obtain the address information of the first GPU 4021 and the second GPU 4022, i.e., the first address information, through the communication connection between the processing unit 401 and the cluster 402, and also obtain the address information of the first virtual resource 4031 to the fourth virtual resource 4034, i.e., the second address information, and determine the association information between the first address information and the second address information based on the number of GPUs in the first state and the number of virtual resources 403, i.e., associate the address of the first virtual resource 4031 to the address of the first GPU 4021, associate the address of the second virtual resource 4032 to the address of the second GPU 4022, and also associate the address of the third virtual resource 4033 and the address of the fourth virtual resource 4034 to the address of the first GPU 4021.

[0154] For example, after the processing unit 401 determines the association information, the processing unit 401 can control the first GPU 4021 and the second GPU 4022 to perform the task processing operation based on the association information through the address of the first virtual resource 4031 and the address of the second virtual resource 4032, so that the task processing operation can be started and performed even if the number of GPUs in the cluster 402 in the first state cannot meet the demand of the task processing operation.

[0155] For example, the processing unit 401 can send the kth processing data to the first virtual resource 4031 to the fourth virtual resource 4034 based on the association information, and after the first virtual resource 4031 obtains the data corresponding to the first GPU 4021 in the kth processing data, the first virtual resource 4031 can send the data to the first GPU 4021, and after the second virtual resource 4032 obtains the data corresponding to the second GPU 4022 in the kth processing data, the second virtual resource 4032 can send the data to the second GPU 4022, so as to execute the kth task processing process through the first GPU 4021 and the second GPU 4022. After the kth task processing process is completed, the first GPU 4021 and the second GPU 4022 can send the processing result data to the processing unit 401 through the first virtual resource 4031 and the second virtual resource 4032, respectively, and the third virtual resource 4033 and the fourth virtual resource 4034 can send preset data such as empty gradient data to the processing unit 401; for example, after the processing unit 401 receives the above processing result data and empty gradient data, the processing unit 401 can process these data to obtain the k+1th processing data, and send the k+1th processing data to the first GPU 4021 and the second GPU 4022 through the first virtual resource 4031 and the second virtual resource 4032 based on the association relationship.

[0156] For example, if the third GPU 4023 in the cluster 402 is detected to switch to the first state, the association information between the address of the third virtual resource 4033 and the address of the first GPU 4021, the association information between the address of the third virtual resource 4033 and the address of the third GPU 4023, and the data corresponding to the first GPU 4021 in the k+1 processing data received by the first GPU 4021 can be sent to the third GPU 4023; at this time, the processing unit 401 can control the first GPU 4021 to the third GPU 4023 to perform the k+1 task processing process based on the association information through the first virtual resource 4031 to the third virtual resource 4033, respectively.

[0157] For example, since the number of GPUs in the first state in the cluster 402 is less than the number of virtual resources 403, the address of the fourth virtual resource 4034 can still be associated with the address of the first GPU 4021; for example, the processing unit 401 can associate the address information of the fourth virtual resource 4034 to the address of the GPU switched to the first state after detecting that there is a GPU in the cluster 402 switched to the first state, and control the GPU and the first GPU 4021 to the third GPU 4023 to continue to perform the task processing process through the method provided in the foregoing embodiments.

[0158] As can be seen from the above, in the task processing method provided in the embodiments of the present application, when the number of GPUs in the first state in the cluster 402 cannot fully meet the conditions of the task processing operation, the dynamic execution of the task processing operation is realized, and during the execution of the task processing operation, the GPU switched to the first state in the cluster 402 can be dynamically, flexibly and in real time added to the task processing process without affecting the normal operation of the GPU already in the running state, thereby improving the flexibility and efficiency of the task processing operation.

[0159] Based on the foregoing embodiments, the embodiments of the present application also provide a task processing device, Figure 5 For the structure schematic diagram of the task processing device 5 provided in the embodiments of the present application, as shown in Figure 5 The device can include:

[0160] The processing unit 501 is configured to control the first resource to perform the corresponding task processing operation through N virtual resource identifiers in response to a task processing instruction; wherein N includes the number of data processing resources required for the task processing operation, and the first resource includes n data processing resources in the first state in the cluster; n is an integer greater than or equal to 1, and n is less than N.

[0161] The processing unit 501 is further configured to, in a case where the second resource is switched from the second state to the first state, associate the second resource with the N-n virtual resource identifiers; the second processing resource includes at least one data processing resource in the cluster except the first resource.

[0162] The processing unit 501 is further configured to control the first resource and the second resource to continue performing the task processing operation through the N virtual resource identifiers.

[0163] In some embodiments, the processing unit 501 is configured to obtain association information; the association information includes mapping information between the N virtual resource identifiers and the data processing resources.

[0164] The processing unit 501 is configured to associate the n virtual resource identifiers to the first resource and the N-n virtual resource identifiers to the third resource based on the association information; the third resource includes any data processing resource in the first resource.

[0165] The processing unit 501 is configured to control the first resource to perform the task processing operation through the N virtual resource identifiers based on the association information.

[0166] The processing unit 501 is configured to obtain first data through the n virtual resource identifiers based on the association information; the first data includes processing result data of the first resource in a kth task processing process; the task processing operation includes multiple task processing processes; k is an integer greater than or equal to 1.

[0167] The processing unit 501 is further configured to obtain second data through the N-n virtual resource identifiers based on the association information; the second data includes preset data.

[0168] The processing unit 501 is configured to process the first data and the second data to obtain k+1 processing data.

[0169] In some embodiments, the processing unit 501 is configured to send the k+1 processing data to the first resource through the n virtual resource identifiers based on the association information, so that the first resource performs a k+1th task processing process based on the k+1 processing data.

[0170] In some embodiments, the processing unit 501 is configured to switch the association information between the target virtual resource identifier and the third resource to association information between the target virtual resource identifier and the second resource, and associate the second resource through the association information between the target virtual resource identifier and the second resource; the target virtual resource identifier includes virtual resource identifiers in the N-n virtual resource identifiers which are same in number as data processing resources in the second resource.

[0171] In some embodiments, the processing unit 501 is configured to transmit third data obtained from the third resource to the second resource, wherein the third data comprises data corresponding to the third resource in the hth processing data obtained by the third resource from the processing unit, and h is an integer greater than or equal to 1.

[0172] The processing unit 501 is further configured to control the second resource to execute the hth task processing procedure based on the third data and control the first resource to execute the hth task processing procedure based on the hth processing data through the target virtual resource identifier.

[0173] In some embodiments, the processing unit 501 is configured to obtain first address information and second address information, and obtain association information based on the first address information and the second address information, wherein the first address information comprises address information of the data processing resource, and the second address information comprises the N virtual resource identifiers.

[0174] In some embodiments, the processing unit 501 is configured to create the N virtual resources based on the task processing instruction, and set virtual resource identifiers for the N virtual resources to obtain the N virtual resource identifiers.

[0175] In some embodiments, the processing unit 501 is configured to adjust the state of the first resource to the second state after controlling the first resource to execute the task processing operation through the N virtual resource identifiers, and adjust the state of the second resource to the second state after controlling the first resource and the second resource to continue executing the task processing operation through the N virtual resource identifiers.

[0176] The processing unit 501 described above can be implemented by a processor of an electronic device, and the processor can be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0177] As can be seen from the above, the task processing apparatus provided by the embodiments of the present application can still execute the task processing operation when the number of data processing resources in the first state in the cluster is less than the number of data processing resources required by the task processing operation, and can dynamically and flexibly associate the available data processing resources to the task processing operation to continue executing the task processing operation when at least one data processing resource in the cluster switches to the first state, thereby realizing flexible execution of the task processing operation when the number of available data processing resources cannot meet the requirements of the task processing operation, dynamic addition of data processing resources during the task processing operation, and greatly improving the flexibility of the task processing operation based on the cluster and the efficiency of the task processing operation.

[0178] Based on the foregoing embodiments, the embodiments of the present application further provide a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor of an electronic device to implement the task processing method according to any one of the foregoing embodiments.

[0179] The foregoing description of various embodiments is intended primarily for purposes of illustration, and the same or similar features can be combined or divided in various embodiments. For simplicity, the same or similar features are not always shown in each diagram, and detailed descriptions of well-known functions and components are omitted to avoid obscuring the subject matter. Reference numbers or characters can be shared among different embodiments.

[0180] The methods disclosed in the various method embodiments of the present application can be combined, if not in conflict, to yield still further method embodiments.

[0181] The features disclosed in the various product embodiments of the present application can be combined, if not in conflict, to yield still further product embodiments.

[0182] The features disclosed in the various method or device embodiments of the present application can be combined, if not in conflict, to yield still further method or device embodiments.

[0183] It should be noted that the computer readable storage medium can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface memory, an optical disc, a Compact Disc Read-Only Memory (CD-ROM), or the like. The computer readable storage medium can also be various electronic devices including one or any combination of the above storage devices, such as a mobile phone, a computer, a tablet device, a personal digital assistant, and the like.

[0184] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a", "comprising", or "comprises" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0185] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0186] Those skilled in the art can clearly understand the above-mentioned embodiment method by means of software and necessary general hardware nodes, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device) to execute the methods described in various embodiments of the present application.

[0187] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0188] These computer program instructions can also be stored in a computer-readable storage medium that can guide the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0189] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0190] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A task processing method, the method comprising: in response to a task processing instruction, a processing unit controls a first resource to perform a task processing operation via N virtual resource identifiers, wherein N comprises a number of data processing resources required by the task processing operation, the first resource comprises n data processing resources in a first state in a cluster, n is an integer greater than or equal to 1, and n is less than N; in a case where a second resource is switched from a second state to the first state, the processing unit associates the second resource via N-n virtual resource identifiers in the N virtual resource identifiers, wherein the second resource comprises at least one data processing resource in the cluster other than the first resource, each virtual resource identifier is associated with a data processing resource, the first state is a state in which at least part of a resource of a data processing resource is available, and the second state is different from the first state; the processing unit controls the first resource and the second resource to continue performing the task processing operation via the N virtual resource identifiers.

2. The method of claim 1, wherein, The processing unit controls a first resource to perform a task processing operation via N virtual resource identifiers, comprising: obtaining association information, wherein the association information comprises mapping information between the N virtual resource identifiers and the data processing resources; the processing unit associates n virtual resource identifiers to the first resource and N-n virtual resource identifiers to a third resource based on the association information, wherein the third resource comprises any data processing resource in the first resource; the processing unit controls the first resource to perform the task processing operation via the N virtual resource identifiers based on the association information.

3. The method of claim 2, wherein, The processing unit controls the first resource to perform the task processing operation via the N virtual resource identifiers based on the association information, comprising: the processing unit obtains first data via the n virtual resource identifiers based on the association information, wherein the first data comprises processing result data of the first resource in a kth task processing process, the task processing operation comprises multiple task processing processes, and k is an integer greater than or equal to 1; the processing unit obtains second data via the N-n virtual resource identifiers based on the association information, wherein the second data comprises preset data; the processing unit processes the first data and the second data to obtain k+1 processing data.

4. The method of claim 3, wherein, The processing unit controls the first resource to perform the task processing operation via the N virtual resource identifiers based on the association information, further comprising: the processing unit sends the k+1 processing data to the first resource via the n virtual resource identifiers based on the association information, so that the first resource performs a k+1th task processing process based on the k+1 processing data.

5. The method of claim 2, wherein, The processing unit associates the second resource via N-n virtual resource identifiers, comprising: The processing unit switches to association information between the target virtual resource identifier and the second resource, and associates the second resource through the association information between the target virtual resource identifier and the second resource; wherein the target virtual resource identifier comprises virtual resource identifiers in the N-n virtual resource identifiers which are same in number as data processing resources in the second resource.

6. The method of claim 5, wherein, The processing unit controls the first resource and the second resource to continue to perform the task processing operation through the N virtual resource identifiers, comprising: transmitting third data obtained from the third resource to the second resource; wherein the third data comprises data corresponding to the third resource in the hth processing data obtained by the third resource from the processing unit; h is an integer greater than or equal to 1; The processing unit controls the second resource to perform the hth task processing process based on the third data through the target virtual resource identifier, and controls the first resource to perform the hth task processing process based on the hth processing data through the n virtual resource identifiers.

7. The method of claim 2, wherein, The obtaining of the association information comprises: obtaining first address information and second address information; wherein the first address information comprises address information of the data processing resources; and the second address information comprises the N virtual resource identifiers; The processing unit obtains the association information based on the first address information and the second address information.

8. The method according to any one of claims 1 to 7, wherein, The method further comprises: The processing unit creates N virtual resources based on the task processing instruction, and sets virtual resource identifiers for the N virtual resources to obtain the N virtual resource identifiers.

9. The method of claim 1, wherein, The method further comprises: After the processing unit controls the first resource to perform the task processing operation through the N virtual resource identifiers, the state of the first resource is adjusted to the second state; and after the processing unit controls the first resource and the second resource to continue to perform the task processing operation through the N virtual resource identifiers, the state of the second resource is adjusted to the second state.

10. A task processing apparatus, comprising: a processing unit configured to control a first resource to perform a corresponding task processing operation through N virtual resource identifiers in response to a task processing instruction; wherein N comprises a number of data processing resources required by the task processing operation, and the first resource comprises n data processing resources in a cluster in a first state; n is an integer greater than or equal to 1, and n is less than N; The processing unit is further configured to associate a second resource through N-n virtual resource identifiers in the N virtual resource identifiers in a case where the second resource is switched from a second state to the first state; wherein the second resource comprises at least one data processing resource in the cluster other than the first resource; each virtual resource identifier is associated with a data processing resource; the first state is a state in which at least part of the data processing resources is available, and the second state is different from the first state. The processing unit is further configured to control the first resource and the second resource to continue performing the task processing operation by using the N virtual resource identifiers.

Images