Task processing method, task execution method, data processing method and device

Abstract

The present disclosure provides a task processing method, a task execution method, a data processing method and a device, which relate to the field of artificial intelligence technology, and in particular to the technical fields of big data, cloud computing, deep learning and distribution. The task processing method includes: in response to detecting a conversion task, a first processing unit obtains original model state information according to a model identifier indicated by the conversion task, wherein the conversion task includes an original parallel strategy identifier and a target parallel strategy identifier; the first processing unit determines the conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier; the first processing unit converts the original model state information based on the conversion strategy to obtain the target model state information; and the first processing unit sends the target model state information to multiple second processing units.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of big data, cloud computing, deep learning and distribution, etc. Specifically, it relates to a task processing method, a task execution method, a data processing method, a device, an electronic device and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the amount of sample data, model structure complexity and number of iterations of neural network models are increasing, and large model technology has emerged. Large models can refer to neural network models that include large-scale parameters. Summary of the invention

[0003] The present disclosure provides a task processing method, a task execution method, a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a task processing method is provided, comprising: in response to detecting a conversion task, a first processing unit obtains original model state information according to a model identifier indicated by the conversion task, wherein the conversion task includes an original parallel strategy identifier and a target parallel strategy identifier; the first processing unit determines a conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier; the first processing unit performs conversion processing on the original model state information based on the conversion strategy to obtain target model state information; and the first processing unit sends the target model state information to multiple second processing units.

[0005] According to another aspect of the present disclosure, a task execution method is provided, including: in response to receiving a target slice from a first processing unit, a second processing unit inputs a sample into a model to obtain output information, wherein the sample includes label information and the model is constructed based on the target slice; the second processing unit determines a loss function value based on the output information and the label information; the second processing unit adjusts the target slice of the model based on the loss function value to obtain a trained model; and the second processing unit sends the adjusted target slice to the first processing unit.

[0006] According to another aspect of the present disclosure, a data processing method is provided, including: a second processing unit obtains data to be processed; and the second processing unit inputs the data to be processed into a trained model to obtain a data processing result, wherein the trained model is obtained using a task processing method.

[0007] According to another aspect of the present disclosure, a task processing device is provided, including: a first acquisition module, used for, in response to detecting a conversion task, a first processing unit to obtain original model state information according to a model identifier indicated by the above conversion task, wherein the above conversion task includes an original parallel strategy identifier and a target parallel strategy identifier; a first determination module, used for the above first processing unit to determine the conversion strategy according to the above original parallel strategy identifier and the above target parallel strategy identifier; a conversion processing module, used for the above first processing unit to perform conversion processing on the above original model state information based on the above conversion strategy to obtain target model state information; and a first sending module, used for the above first processing unit to send the above target model state information to multiple second processing units.

[0008] According to another aspect of the present disclosure, a task execution device is provided, including: a first input module, used for, in response to receiving a target slice from a first processing unit, a second processing unit inputting a sample into a model to obtain output information, wherein the sample includes label information and the model is constructed based on the target slice; a second determination module, used for the second processing unit to determine a loss function value based on the output information and the label information; an adjustment module, used for the second processing unit to adjust the target slice of the model based on the loss function value to obtain a trained model; and a second sending module, used for the second processing unit to send the adjusted target slice to the first processing unit.

[0009] According to another aspect of the present disclosure, a data processing device is provided, comprising: a second acquisition module, used for the second processing unit to acquire data to be processed; and a second input module, used for the second processing unit to input the above-mentioned data to be processed into a trained model to obtain a data processing result, wherein the above-mentioned trained model is obtained using a task processing device.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the present disclosure.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method described in the present disclosure.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 Schematically illustrates an exemplary system architecture to which a task processing method, a task execution method, a data processing method and an apparatus according to an embodiment of the present disclosure can be applied;

[0016] Figure 2 A flowchart schematically shows a task processing method according to an embodiment of the present disclosure;

[0017] Figure 3 An example schematic diagram schematically shows a process of obtaining original model state information according to an embodiment of the present disclosure;

[0018] Figure 4A An example schematic diagram of a process of determining a conversion strategy when the original parallel strategy includes the first parallel strategy and the target parallel strategy includes the first parallel strategy according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 4B An example schematic diagram of a process of determining a conversion strategy in a case where the original parallel strategy includes the second parallel strategy and the target parallel strategy includes the second parallel strategy according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 5A An example schematic diagram of a process of converting original model state information to obtain intermediate model state information based on a first conversion strategy according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 5B An example schematic diagram of a process of converting the intermediate model state information to obtain the target model state information based on the second conversion strategy according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 6 An example schematic diagram schematically shows a task processing process according to another embodiment of the present disclosure;

[0023] Fig. 7A An example schematic diagram schematically shows a process in which a first processing unit sends target model state information to a plurality of second processing units according to an embodiment of the present disclosure;

[0024] Figure 7BAn example schematic diagram schematically shows a task processing process according to another embodiment of the present disclosure;

[0025] Figure 8 A flowchart of a task execution method according to an embodiment of the present disclosure is schematically shown;

[0026] Fig. 9 An example schematic diagram schematically shows a task execution process according to an embodiment of the present disclosure;

[0027] Fig.10 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0028] Fig.11 A block diagram of a task processing device according to an embodiment of the present disclosure is schematically shown;

[0029] Fig.12 A block diagram of a task execution device according to an embodiment of the present disclosure is schematically shown;

[0030] Fig.13 A block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0031] Fig.14 A block diagram of an electronic device suitable for implementing a task processing method, a task execution method, and a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] Since the communication between a single accelerator load and different accelerator loads is an urgent problem to be solved when the model is large or the sample data scale is large. In one example, a distributed parallel strategy based on multi-dimensional mixing can be used to achieve large-scale training for large models. For example, the distributed parallel strategy can include at least one of the following: tensor parallel strategy, pipeline parallel strategy, slice parallel strategy and data parallel strategy.

[0034] In each of the above-mentioned distributed parallel strategies, the model state information is segmented and saved on different processing devices, and in the process of saving the model state information, each processing device only saves the part of the model state information that is segmented. Since different distributed parallel strategies have different segmentation methods, the storage formats of different parts of the model state information are also different. During the model training process, it is often necessary to change the distributed parallel strategy. In this case, since the model state information corresponding to the original distributed parallel strategy cannot be directly hot-started and loaded by the changed distributed parallel strategy, it is necessary to convert the model state information between different distributed parallel strategies.

[0035] In one example, an offline script-based conversion method can be used to achieve the conversion of model state information. However, since the offline script-based conversion method is completed in the offline process rather than in the model hot start process, the conversion process consumes more machine resources and the conversion time cost is high. In addition, since the offline script-based conversion method requires the development of offline scripts for different models and different distributed parallel strategies, the versatility of the conversion method is poor and it cannot be applied to different models and different distributed parallel strategies.

[0036] To this end, the embodiment of the present disclosure proposes a task processing scheme. For example, in response to detecting a conversion task, the first processing unit obtains the original model state information according to the model identifier indicated by the conversion task, wherein the conversion task includes the original parallel strategy identifier and the target parallel strategy identifier; the first processing unit determines the conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier; the first processing unit performs conversion processing on the original model state information based on the conversion strategy to obtain the target model state information; and the first processing unit sends the target model state information to multiple second processing units.

[0037] According to an embodiment of the present disclosure, the first processing unit can obtain the original model state information according to the model identifier specified by the conversion task, and convert the original model state information based on the conversion strategy determined according to the original parallel strategy identifier and the target parallel strategy identifier to obtain the target model state information, thereby realizing adaptive online conversion of the model state information when the parallel strategy changes, reducing the conversion cost in the model parallel strategy conversion process, improving the versatility of the parallel strategy conversion, and thereby improving the efficiency of task processing.

[0038] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] In the technical solution of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0040] Figure 1 An exemplary system architecture to which a task processing method, a task execution method, a data processing method and an apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0041] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the task processing method, task execution method, data processing method and device can be applied may include a master server, but the master server may implement the task processing method, task execution method, data processing method and device provided by the embodiments of the present disclosure without interacting with a slave server.

[0042] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a master server 101, a slave server 102, a slave server 103, and a network 104. The network 104 is used to provide a medium for communication links between the master server 101 and the slave servers 102 and 103. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0043] The main server 101 can be set in an electronic device, for example, and the electronic device can be a computer. The main server 101 can be various types of servers that provide various services. For example, the main server 101 can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Servers). The main server 101 can also be a server of a distributed system, or a server combined with a blockchain.

[0044] The main server 101 may include a first processing unit 101_1 and a storage unit 101_2. The first processing unit 101_1 may be a processing unit with computing power and storage power, for example, a central processing unit (CPU) in the main server. The storage unit 101_2 may be used to store at least one candidate model identifier and model state information corresponding to each candidate model identifier.

[0045] The slave server 102 and / or the slave server 103 may be, for example, arranged in an electronic device, which may be, for example, a computer. The slave server 102 may include a second processing unit 102_1, a second processing unit 102_2, ..., and a second processing unit 102_M. M may be a positive integer. The slave server 103 may include a second processing unit 103_1, a second processing unit 103_2, ..., and a second processing unit 103_N. N may be a positive integer. The second processing unit 102_1, the second processing unit 102_2, ..., and the second processing unit 102_M and / or the second processing unit 103_1, the second processing unit 103_2, ..., and the second processing unit 103_N may be a processing unit having computing power and storage capacity, such as a central processing unit, a graphics processing unit (GPU), and a neural network processing unit (NNP).

[0046] It should be noted that the task processing method provided in the embodiment of the present disclosure can generally be executed by the first processing unit 101_1 included in the master server 101. Alternatively, the task processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the first processing unit 101_1 included in the master server 101 and can communicate with the slave server 102 and / or the slave server 103.

[0047] The task execution method and / or data processing method provided in the embodiments of the present disclosure may generally be executed by the second processing unit 102_1, the second processing unit 102_2, ..., the second processing unit 102_M included in the slave server 102 and / or the second processing unit 103_1, the second processing unit 103_2, ..., the second processing unit 103_N included in the slave server 103. Alternatively, the task execution method and / or data processing method provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the second processing unit 102_1, the second processing unit 102_2, ..., the second processing unit 102_M and / or the second processing unit 103_1, the second processing unit 103_2, ..., the second processing unit 103_N and that is capable of communicating with the master server 101.

[0048] It should be understood that Figure 1 The number of master servers, slave servers, first processing units, second processing units, and networks in the embodiment is only illustrative. Any number of master servers, slave servers, first processing units, second processing units, and networks may be provided as required.

[0049] Figure 2The flowchart of the task processing method according to the embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the task processing method 200 may include operations S210 to S240.

[0051] In operation S210 , in response to detecting a conversion task, the first processing unit obtains original model state information according to a model identifier indicated by the conversion task, wherein the conversion task includes an original parallel strategy identifier and a target parallel strategy identifier.

[0052] In operation S220, the first processing unit determines a conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier.

[0053] In operation S230, the first processing unit performs conversion processing on the original model state information based on the conversion strategy to obtain the target model state information.

[0054] In operation S240 , the first processing unit sends the target model state information to the plurality of second processing units.

[0055] The first processing unit may refer to a processing unit in the master server that has computing power and storage capacity. For example, the first processing unit may include a central processing unit in the master server. The second processing unit may refer to a processing unit in the slave server that has computing power and storage capacity. For example, the second processing unit may include at least one of the following: a graphics processing unit, a central processing unit, and a neural network processing unit in different slave servers, or different graphics processing units, central processing units, and neural network processing units in the same slave server.

[0056] The first processing unit and the second processing unit can communicate with each other. For example, the first processing unit can be used to execute the task processing method provided by the embodiment of the present disclosure, that is, to convert the original model state information to obtain the target model state information, and send the target model state information to the second processing unit. The second processing unit can be used to execute the task execution method provided by the embodiment of the present disclosure, that is, to build a model according to the target slice received from the first processing unit, train the model to obtain the trained model and the adjusted target slice, and send the adjusted target slice to the first processing unit.

[0057] The conversion task may include a model identifier, an original parallel strategy identifier, and a target parallel strategy identifier. The model identifier may be used to uniquely identify the model. After detecting the conversion task, the first processing unit may obtain the original model state information corresponding to the model identifier according to the model identifier. The original model state information may refer to the model state information obtained by training using the original parallel strategy. For example, the model state information may include at least one of the following: a model parameter and an optimizer state parameter.

[0058] The original model state information may be obtained by the first processing unit from the storage unit. The storage unit may refer to a processing unit with storage capability in the main server. The storage unit may store at least one candidate model identifier obtained by pre-training and candidate original model state information corresponding to each candidate model identifier.

[0059] The original parallel strategy identifier may be used to identify the original parallel strategy. The original parallel strategy may refer to the distributed parallel strategy adopted by the model corresponding to the model identifier before the transformation task is executed. The target parallel strategy identifier may be used to identify the target parallel strategy. The target parallel strategy may refer to the distributed parallel strategy adopted by the model corresponding to the model identifier after the transformation task is executed.

[0060] After the conversion task is detected, the conversion strategy (i.e., Placement) can be determined based on the original parallel strategy identifier and the target parallel strategy identifier. The conversion strategy can be used to automatically convert the original model state information so as to obtain the target model state information adapted to the target parallel strategy. For example, the conversion strategy may include at least one of the following: a processing strategy, an aggregation strategy, and a splitting strategy. The processing strategy may refer to a strategy for rearranging at least one slice. The aggregation strategy may refer to a strategy for performing an aggregation operation on at least one slice. The splitting strategy may refer to a strategy for performing a splitting operation on the model state information to obtain at least one slice.

[0061] The target model state information may refer to model state information obtained by training using a target parallel strategy. After obtaining the target model state information, the first processing unit may send the target model state information to multiple second processing units. For example, the target model state information may include at least one target slice, and the at least one target slice may be grouped to obtain a target slice group corresponding to the number of second processing units. On this basis, each target slice group may be sent to the second processing unit respectively.

[0062] According to an embodiment of the present disclosure, the first processing unit can obtain the original model state information according to the model identifier specified by the conversion task, and convert the original model state information based on the conversion strategy determined according to the original parallel strategy identifier and the target parallel strategy identifier to obtain the target model state information, thereby realizing adaptive online conversion of the model state information when the parallel strategy changes, reducing the conversion cost in the model parallel strategy conversion process, improving the versatility of the parallel strategy conversion, and thereby improving the efficiency of task processing.

[0063] Figure 3 An example schematic diagram of a process for obtaining original model state information according to an embodiment of the present disclosure is schematically shown.

[0064] like Figure 3 As shown in 300, the main server 301 may include a first processing unit 301_1 and a storage unit 301_2. The storage unit 301_2 may refer to a processing unit in the main server 301 with storage capability.

[0065] The storage unit 301_2 may store at least one candidate model identifier obtained by pre-training and candidate original model state information corresponding to each candidate model identifier. For example, at least one candidate model identifier may include candidate model identifier 305_1, candidate model identifier 305_2, ..., candidate model identifier 305_P, and at least one candidate original model state information may include candidate original model state information 306_1 corresponding to candidate model identifier 305_1, candidate original model state information 306_2 corresponding to candidate model identifier 305_2, ..., candidate original model state information 306_P corresponding to candidate model identifier 305_P. P is a positive integer.

[0066] In one example, before executing the task processing method provided by the present disclosure, an event listener 302 may also be pre-set. The configuration information of the event listener 302 may include a listening element and a listening type. In one example, the listening element may include at least one of the following: the number of second processing units corresponding to each of the at least one candidate model identifier and the content of the model state information. The listening type may include a quantity change event and a content change event. In this case, the event listener 302 may be used to listen to a preset event corresponding to each of the at least one candidate model identifier.

[0067] The preset events can be set according to actual business needs and are not limited here. The preset events may include at least one of the following: a change event in the number of second processing units and a change event in the content of the model state information. For example, a change event in the number of second processing units may refer to an increase or decrease in the number of second processing units. Alternatively, a change event in the content of the model state information may refer to a change in a model hyperparameter, for example, a hyperparameter may include a batch size and / or a sequence length.

[0068] In response to the event listener 302 being triggered, that is, after the event listener 302 listens to the preset event, the first processing unit 301_1 can automatically generate a conversion task 303 according to the model identifier 304 corresponding to the preset event. The conversion task 303 can refer to a task for automatically converting the original model state information corresponding to the model identifier 304 into the target model state information.

[0069] According to an embodiment of the present disclosure, by setting an event listener, a preset event corresponding to at least one candidate model identifier can be listened to. On this basis, when the event listener is triggered, the first processing unit can automatically respond and generate a corresponding conversion task according to the listened event and the corresponding model identifier, thereby reducing manual intervention and improving the intelligence and automation of the task processing process.

[0070] In one example, after the first processing unit 301_1 detects the generated conversion task 303, it can match the model identifier 304 indicated by the conversion task 303 with at least one candidate model identifier stored in the storage unit 301_2, namely, candidate model identifier 305_1, candidate model identifier 305_2, ..., candidate model identifier 305_P, respectively, so as to determine the target model identifier that matches the model identifier 304, and obtain the candidate original model state information corresponding to the target model identifier.

[0071] According to the embodiments of the present disclosure, by storing the candidate model identifiers and the original model state information in a one-to-one correspondence in a storage unit, it is beneficial to manage and maintain the model state information, and facilitate the subsequent task processing process. On this basis, when a conversion task is detected, according to the specified model identifier, the candidate original model state information corresponding to the model identifier in the storage unit is accessed and obtained, thereby improving the efficiency and accuracy of obtaining the original model state information.

[0072] In one example, the original model state information may include original model parameters and original optimizer state parameters. Model parameters may refer to adjustable variables in a machine learning or deep learning model that can be used to control the behavior and performance of the model. For example, the original model parameters may include at least one of the following: weights, biases, learning rate, batch size, regularization parameters, and activation function parameters.

[0073] The optimizer state parameters may refer to the state information maintained by the optimizer during the training process that can be used to help the optimizer track and adjust the update of model parameters. Alternatively, the original optimizer state parameters may include at least one of the following: momentum, learning rate decay, accumulated gradients, optimizer state vectors, and regularization parameters state.

[0074] Figure 4A An example schematic diagram of a process of determining a conversion strategy when the original parallel strategy includes the first parallel strategy and the target parallel strategy also includes the first parallel strategy according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 4A As shown, in 400A, the first parallel strategy 401 may include at least one of the following: a tensor parallel strategy (Tensor Parallelism, TP) 4011 and a first type of slicing parallel strategy (Sharding V1 Parallelism, Sharding V1) 401_2.

[0076] For example, the tensor parallel strategy 4011 can be understood as a strategy for splitting the model's tensor (i.e., Tensor) into different second processing units to implement large-scale model training. Alternatively, the first type of slice parallel strategy 4012 can be understood as a strategy for splitting the model's operator (i.e., Operator) into different second processing units to implement large-scale model training.

[0077] When the tensor parallel strategy 4011 and / or the first type of slice parallel strategy 4012 are adopted, it is necessary to perform internal segmentation of the model parameters, and segment the same model parameters to different second processing units for storage and calculation. In addition, it is also necessary to perform internal segmentation of the optimizer state parameters, and segment the same optimizer state parameters to different second processing units for storage and calculation. In view of this, when performing parallel strategy conversion, it is necessary to solve the problems of how to re-aggregate the same parameter after segmentation, how to re-segment the same parameter after re-aggregation, and how to rearrange the re-aggregated or re-segmented parameters.

[0078] In one example, a first conversion strategy 403 (i.e., Restore1) can be determined based on the original parallel strategy identifier. The first conversion strategy 403 can refer to a strategy for converting the original model state information into the intermediate model state information. In addition, a second conversion strategy 404 (i.e., Restore2) can be determined based on the target parallel strategy identifier. The second conversion strategy 404 can refer to a strategy for converting the intermediate model state information into the target model state information.

[0079] In the case where the original parallel strategy corresponding to the original parallel strategy identifier includes the first parallel strategy and / or the target parallel strategy corresponding to the target parallel strategy identifier includes the first parallel strategy, a conversion mechanism (i.e., Replacement) of general model state information after the parallel strategy is changed can be configured according to the conversion strategies corresponding to different parallel strategy identifiers, i.e., for different parallel strategy identifiers. In addition, the parallel strategy conversion layer can be further encapsulated for different parallel strategy identifiers, so as to directly call the corresponding parallel strategy conversion layer according to the parallel strategy identifier to perform the conversion operation of the model state information.

[0080] According to the embodiment of the present disclosure, by determining the first conversion strategy according to the original parallel strategy identifier, and determining the second conversion strategy according to the target parallel strategy identifier, the conversion strategy is automatically determined based on the parallel strategy identifier, thereby improving the accuracy of the conversion strategy determination. On this basis, by converting the original model state information into the intermediate model state information based on the first conversion strategy, and converting the intermediate model state information into the target model state information based on the second conversion strategy, it is beneficial to improve the efficiency of subsequent task processing based on the conversion strategy.

[0081] In one example, the original model state information may include at least one original slice. Each original slice may correspond to an information item identifier and an original slice identifier. The information item identifier may be used to uniquely identify different model parameter items or optimizer state parameter items. The original slice identifier may be used to uniquely identify different original slices.

[0082] When the original parallel strategy corresponding to the original parallel strategy identifier includes the first parallel strategy 401, the first conversion strategy 403 may include the first processing strategy 4031 and the aggregation strategy 4032. For example, the first processing strategy 4031 and the aggregation strategy 4032 may be determined according to the original parallel strategy identifier.

[0083] The first processing strategy 4031 may refer to a strategy for performing a rearrangement operation (i.e., Reshard) on at least one original slice. For example, the rearrangement operation may refer to filtering out original slices belonging to the same information item identifier according to the information item identifier corresponding to each original slice, and obtaining original slice groups corresponding to each information item identifier. Each original slice group includes at least one original slice. On this basis, for the original slice groups corresponding to each information item identifier, at least one original slice in the original slice group may be arranged according to the original slice identifier corresponding to each original slice in the original slice group, and at least one arranged original slice corresponding to each information item identifier may be obtained.

[0084] In one example, the first processing strategy 4031 may be implemented using a first application programming interface. The first application programming interface may implement rearrangement of operation logic by passing in a first callback function (i.e., filter func). The first callback function may be used to provide routing information to determine whether a parameter or a slice is to be allocated to a second processing unit.

[0085] Aggregation strategy 4032 may refer to a strategy for performing an aggregation operation (i.e., Merge) on at least one original slice arranged corresponding to each information item identifier. For example, the aggregation operation may refer to aggregating all the original slices that have been segmented by internal parameters for at least one original slice arranged corresponding to each information item identifier, thereby constructing an intermediate slice corresponding to the information item identifier. The intermediate slice may refer to a complete slice that has not been segmented by internal parameters.

[0086] In one example, the aggregation strategy 4032 may be implemented using a second application programming interface. The second application programming interface may implement the aggregation operation logic by passing in a second callback function (ie, merge_func). The second callback function may be used to aggregate slices belonging to the same parameter.

[0087] According to an embodiment of the present disclosure, when the original parallel strategy corresponding to the original parallel strategy identifier includes a first parallel strategy, since the first processing strategy and the aggregation strategy are automatically determined based on the original parallel strategy identifier, it is beneficial to subsequently flexibly process and convert the original model state information based on the first processing strategy and the aggregation strategy, thereby improving the efficiency and accuracy of task processing.

[0088] In one example, when the target parallel strategy corresponding to the target parallel strategy identifier includes the first parallel strategy 401, the second conversion strategy 404 may include a split strategy 4041 and a second processing strategy 4042. For example, the split strategy 4041 and the second processing strategy 4042 may be determined according to the target parallel strategy identifier.

[0089] The splitting strategy 4041 may refer to a strategy for performing a splitting operation (i.e., Split) on the intermediate model state information. The intermediate model state information may include intermediate slices corresponding to at least one information item identifier. For example, for the intermediate slice corresponding to each information item identifier, the intermediate slice may be split by internal parameters to obtain at least one target slice corresponding to the information item identifier.

[0090] In one example, the splitting strategy 4041 may be implemented using a third application programming interface. The third application programming interface may implement the splitting operation logic by passing in a third callback function (i.e., split_func). The third callback function may be used to split a parameter into multiple slices.

[0091] The second processing strategy 4042 may refer to a strategy for performing a rearrangement operation (i.e., Resharding) on ​​at least one target slice that has been segmented and corresponds to each information item identifier. For example, for at least one target slice that has been segmented and corresponds to each information item identifier, at least one target slice that corresponds to each second processing unit identifier may be determined based on the second processing unit identifier that corresponds to each second processing unit.

[0092] According to an embodiment of the present disclosure, when the target parallel strategy corresponding to the target parallel strategy identifier includes a first parallel strategy, since the splitting strategy and the second processing strategy are automatically determined based on the target parallel strategy identifier, it is beneficial to subsequently flexibly process and convert the intermediate model state information based on the splitting strategy and the second processing strategy, thereby improving the efficiency and accuracy of task processing.

[0093] Figure 4B An example schematic diagram of a process of determining a conversion strategy when the original parallel strategy includes the second parallel strategy and the target parallel strategy includes the second parallel strategy according to an embodiment of the present disclosure is schematically shown.

[0094] like Figure 4B As shown, in 400B, the second parallel strategy 402 may include at least one of the following: a pipeline parallel strategy (Pipeline Parallel, PP) 4021 and a second type of slicing parallel strategy (Sharding V2 Parallelism, Sharding V2) 4022.

[0095] For example, the pipeline parallel strategy 4021 can be understood as placing different network layers (i.e., Layer) of the model into different second processing units to implement a strategy for large-scale model training. Alternatively, the second type of slice parallel strategy 4022 can be understood as allocating different operators of the model to different second processing units to implement a strategy for large-scale model training.

[0096] When the pipeline parallel strategy 4021 and / or the second type of slice parallel strategy 4022 are adopted, it is not necessary to segment the model parameters or optimizer parameters internally, but different model parameters and / or different optimizer parameters are allocated to different second processing units for storage and calculation. In view of this, when performing parallel strategy conversion, it is necessary to solve the problem of how to rearrange different parameters.

[0097] In one example, when the original parallel strategy corresponding to the original parallel strategy identifier includes the second parallel strategy 402 and the target parallel strategy includes the second parallel strategy 402, the third conversion strategy 405 can be determined according to the original parallel strategy identifier and the target parallel strategy identifier. The third conversion strategy 405 can include a third processing strategy 4051.

[0098] The third processing strategy 4051 may refer to a strategy for performing a rearrangement operation (i.e., Reshard) on at least one original slice included in the original model state information to obtain at least one target slice included in the target model state information. For example, the rearrangement operation may refer to filtering out original slices belonging to the same information item identifier according to the information item identifier corresponding to each original slice, to obtain an original slice group corresponding to each information item identifier. Each original slice group includes at least one original slice. On this basis, for the original slice group corresponding to each information item identifier, at least one original slice in the original slice group may be arranged according to the original slice identifier corresponding to each original slice in the original slice group, to obtain at least one arranged target slice corresponding to each information item identifier.

[0099] In the case where the original parallel strategy corresponding to the original parallel strategy identifier includes the second parallel strategy and the target parallel strategy corresponding to the target parallel strategy identifier includes the second parallel strategy, a conversion mechanism (i.e., Replacement) of the general model state information after the parallel strategy is changed can be configured according to the conversion strategies corresponding to the different parallel strategy identifiers, i.e., for different parallel strategy identifiers. In addition, the parallel strategy conversion layer can be further encapsulated for different parallel strategy identifiers, so as to directly call the corresponding parallel strategy conversion layer according to the parallel strategy identifier to perform the conversion operation of the model state information.

[0100] According to an embodiment of the present disclosure, when the original parallel strategy corresponding to the original parallel strategy identifier includes the second parallel strategy and the target parallel strategy includes the second parallel strategy, since the third processing strategy is automatically determined based on the original parallel processing identifier and the target parallel strategy identifier, it is beneficial to the subsequent flexible processing and conversion of the original model state information based on the third processing strategy, thereby improving the efficiency and accuracy of task processing.

[0101] Figure 5A An example schematic diagram of a process of converting original model state information to obtain intermediate model state information based on a first conversion strategy according to an embodiment of the present disclosure is schematically shown.

[0102] like Figure 5A As shown, in 500A, after obtaining the original model state information, the original model state information can be converted based on the first conversion strategy (i.e., Restore1) to obtain the intermediate model state information. For example, the original model state information can include the sub-original model state information 501 corresponding to the second processing unit 1 and the sub-original model state information 502 corresponding to the second processing unit 2. The sub-original model state information 501 can include original slices 5030, 5032, 5040, and 5042. The sub-original model state information 502 can include original slices 5031, 5033, 5041, and 5043.

[0103] For each of the above original slices, the number before the underscore in each original slice identifier can be understood as the information item identifier, and the number after the underscore in each original slice identifier can be understood as the original slice identifier corresponding to the information item identifier. For example, original slice 5030 can refer to the first original slice corresponding to the information item identifier "503", and original slice 5041 can refer to the second original slice corresponding to the information item identifier "504".

[0104] According to an embodiment of the present disclosure, since the first conversion strategy is automatically determined according to the original parallel strategy identifier, the accuracy of obtaining the intermediate model state information is improved by converting the original model state information based on the first conversion strategy.

[0105] In one example, the first conversion strategy may include a first processing strategy. Based on the first processing strategy, at least one original slice may be rearranged to obtain at least one arranged original slice. In addition, the first conversion strategy may also include an aggregation strategy. Based on the aggregation strategy, at least one arranged original slice may be aggregated to obtain intermediate model state information.

[0106] According to an embodiment of the present disclosure, by using the first processing strategy to rearrange at least one original slice, the continuity of the obtained arranged at least one original slice can be guaranteed. On this basis, by using the aggregation strategy to aggregate at least one arranged original slice, the accuracy of the intermediate model state information can be improved.

[0107] In one example, at least one original slice to be arranged may be determined in at least one original slice according to the information item identifier of each original slice. For example, according to the information item identifier "503", it may be determined in at least one original slice that the original slices to be arranged corresponding to the information item identifier "503" include original slices 5030, original slices 5032, original slices 5031, and original slices 5033. According to the information item identifier "504", it may be determined in at least one original slice that the original slices to be arranged corresponding to the information item identifier "504" include original slices 5040, original slices 5042, original slices 5041, and original slices 5043.

[0108] On this basis, at least one original slice to be arranged may be rearranged according to the original slice identifier of each original slice to be arranged to obtain at least one arranged original slice corresponding to the information item identifier.

[0109] For example, for at least one original slice to be arranged corresponding to the information item identifier "503", the at least one original slice to be arranged can be rearranged according to the original slice identifier of each original slice to be arranged to obtain the original slice 5030->original slice 5031->original slice 5032->original slice 5033.

[0110] Alternatively, for at least one original slice to be arranged corresponding to the information item identifier "504", the at least one arranged original slice corresponding to the information item identifier "504" can be rearranged according to the original slice identifier of each original slice to be arranged to obtain original slice 5040->original slice 5041->original slice 5042->original slice 5043.

[0111] On this basis, at least one of the arranged original slices can be aggregated according to the information item identifier to obtain the intermediate model state information corresponding to the information item identifier. For example, according to the information item identifier "503", the original slice 503_0->original slice 503_1->original slice 503_2->original slice 503_3 can be aggregated to obtain the intermediate model state information 505 corresponding to the information item identifier "503". Alternatively, according to the information item identifier "504", the original slice 504_0->original slice 504_1->original slice 504_2->original slice 504_3 can be aggregated to obtain the intermediate model state information 506 corresponding to the information item identifier "504".

[0112] According to the embodiments of the present disclosure, since at least one arranged original slice is obtained by sorting the original slices to be arranged corresponding to different information item identifiers according to the original slice identifiers, and the original slices to be arranged are obtained by matching the information item identifiers of the original slices, it is possible to ensure that the original slices with the same information item identifier are associated and arranged in sequence. On this basis, by aggregating at least one arranged original slice according to the information item identifier, the accuracy of the intermediate model state information corresponding to the information item identifier can be improved.

[0113] Figure 5B An example schematic diagram of a process of converting the intermediate model state information to obtain the target model state information based on the second conversion strategy according to an embodiment of the present disclosure is schematically shown.

[0114] like Figure 5B As shown, in 500B, after obtaining the intermediate model state information 507 corresponding to the information item identifier "1" and the intermediate model state information 508 corresponding to the information item identifier "2", the intermediate model state information 507 and the intermediate model state information 508 can be converted and processed respectively based on the second conversion strategy (i.e., Restore2) to obtain the target model state information.

[0115] According to an embodiment of the present disclosure, since the second conversion strategy is automatically determined according to the target parallel strategy identifier, the accuracy of obtaining the target model state information is improved by converting the intermediate model state information based on the second conversion strategy.

[0116] In one example, the second conversion strategy may include a segmentation strategy. Based on the segmentation strategy, the intermediate model state information may be segmented to obtain at least one intermediate slice. In addition, the second conversion strategy may include a second processing strategy. Based on the second processing strategy, at least one intermediate slice may be rearranged to obtain the target model state information.

[0117] According to the embodiments of the present disclosure, by using the segmentation strategy to segment the intermediate model state information, the continuity of at least one intermediate slice can be guaranteed. On this basis, by using the second processing strategy to rearrange at least one intermediate slice, the intermediate model state information can be properly processed and converted according to the requirements, so that the obtained target model state information is compatible with the target parallel strategy identification, which is conducive to improving the efficiency of task processing and subsequent task execution.

[0118] In one example, the segmentation strategy may include the number of second computing units. For each information item identifier, the intermediate model state information corresponding to the information item identifier may be segmented according to the number of second computing units to obtain at least one intermediate slice corresponding to the information item identifier.

[0119] For example, when the number of second computing units is 2, for information item identification "1", the intermediate model state information 507 corresponding to the information item identification "1" may be divided into two, and intermediate slices 5070 and 5071 corresponding to the information item identification "1" are obtained. Alternatively, for information item identification "2", the intermediate model state information 508 corresponding to the information item identification "2" may be divided into two, and intermediate slices 5080 and 5081 corresponding to the information item identification "2" are obtained.

[0120] On this basis, based on the second processing strategy, the middle slices 5070 and 5071 corresponding to the information item identifier "1" and the middle slices 5080 and 5081 corresponding to the information item identifier "2" can be rearranged to obtain the target model state information. For example, the middle slices 5070 and 5080 can be determined as the target slice 509, and the middle slices 5071 and 5081 can be determined as the target slice 510.

[0121] According to the embodiments of the present disclosure, since at least one intermediate slice is obtained by segmenting the intermediate model state information corresponding to each information item identifier, it is possible to ensure that the intermediate slice is compatible with the target parallel strategy, which is conducive to improving the accuracy of the target model state information obtained based on the intermediate slice. In addition, the segmented intermediate slices can also facilitate distributed processing and parallel computing, further improving the scalability and flexibility of task processing.

[0122] Figure 6 An example schematic diagram of a task processing process according to another embodiment of the present disclosure is schematically shown.

[0123] like Figure 6 As shown, in 600, the original model state information may include sub-original model state information 601 corresponding to the second processing unit 3 and sub-original model state information 602 corresponding to the second processing unit 4. The sub-original model state information 601 may include an original slice 6030 and an original slice 6042. The sub-original model state information 602 may include an original slice 6051 and an original slice 6063. For each of the above original slices, the number before the underscore in each original slice identifier may be understood as an information item identifier, and the number after the underscore in each original slice identifier may be understood as an original slice identifier corresponding to the information item identifier.

[0124] In one example, at least one original slice may be rearranged based on the third processing strategy to obtain target model state information. For example, the original slice 603_0, the original slice 604_2, the original slice 605_1, and the original slice 606_3 may be rearranged to obtain a target slice 607 corresponding to the second processing unit 3 including the original slice 603_0 and the original slice 605_1, and a target slice 608 corresponding to the second processing unit 4 including the original slice 604_2 and the original slice 606_3.

[0125] According to an embodiment of the present disclosure, by rearranging at least one original slice based on the third processing strategy, target model state information that is compatible with the target parallel strategy can be obtained, which is beneficial to improving the accuracy of task processing and subsequent task execution.

[0126] Fig. 7A An example schematic diagram schematically shows a process in which a first processing unit sends target model state information to multiple second processing units according to an embodiment of the present disclosure.

[0127] like Fig. 7A As shown, the target model state information may include the number of target slice groups of the second processing units. In 700A, the number of the second processing units is Q, and the target model state information may include Q target slice groups.

[0128] A mapping relationship between the number of target slice groups and the second processing units of the second computing unit may be determined. Each target slice group may include at least one target slice. For example, the mapping relationship may include a one-to-one correspondence between the target slice group 702_1 and the second processing unit 703_1, a one-to-one correspondence between the target slice group 702_2 and the second processing unit 703_2, a one-to-one correspondence between the target slice group 702_3 and the second processing unit 703_3, ..., a one-to-one correspondence between the target slice group 702_Q and the second processing unit 703_Q. Q is a positive integer.

[0129] On this basis, the number of target slice groups of the second computing units can be synchronously sent to multiple second processing units according to the mapping relationship. For example, the first processing unit 701_1 can send the target slice group 702_1 to the second processing unit 703_1, send the target slice group 702_2 to the second processing unit 703_2, send the target slice group 702_3 to the second processing unit 703_3, ..., and send the target slice group 702_Q to the second processing unit 703_Q according to the mapping relationship.

[0130] According to the embodiments of the present disclosure, by determining the one-to-one correspondence between the target slice group and the plurality of second processing units according to the number of second computing units, each target slice group has a corresponding second processing unit, thereby achieving effective mapping between the target slice group and the second processing unit. On this basis, by synchronously sending the target slice groups of the number of second computing units to the corresponding second processing units according to the determined mapping relationship, it is possible to ensure that each second processing unit can obtain the target slice group it needs, which is beneficial for each subsequent second processing unit to use the target slice group for distributed parallel training, thereby improving the efficiency of subsequent task execution.

[0131] Figure 7B An example schematic diagram of a task processing process according to another embodiment of the present disclosure is schematically shown.

[0132] like Figure 7B As shown, in 700B, the first processing unit 704_1 may receive adjusted target slices from a plurality of second processing units. For example, the first processing unit 704_1 may receive an adjusted target slice 706_1 from the second processing unit 705_1, an adjusted target slice 706_2 from the second processing unit 705_2, an adjusted target slice 706_3 from the second processing unit 705_3, ..., an adjusted target slice 706_S from the second processing unit 705_S. S is a positive integer.

[0133] After receiving multiple adjusted target slices, the first processing unit 704_1 can determine the adjusted target model state information based on the multiple adjusted target slices, i.e., adjusted target slice 706_1, adjusted target slice 706_2, adjusted target slice 706_3, ..., adjusted target slice 706_S, and use the adjusted target model state information to update the original model state information stored in the storage unit 704_2.

[0134] According to an embodiment of the present disclosure, a plurality of second processing units send adjusted target slices to a first processing unit, and the first processing unit can receive and integrate these target slices according to a predetermined mapping relationship, thereby obtaining complete adjusted target model state information. On this basis, the first processing unit updates the original model state information stored in the storage unit by using the adjusted target model state information so that the original model state information in the storage unit maintains consistency with the adjusted target model state information, and can reflect the latest state of the model corresponding to the model identifier.

[0135] This is only an exemplary embodiment, but is not limited thereto, and may also include other task processing methods known in the art as long as the efficiency of task processing can be improved.

[0136] Figure 8 The flowchart of the task execution method according to the embodiment of the present disclosure is schematically shown.

[0137] like Figure 8 As shown, the task execution method 800 may include operations S810 to S840.

[0138] In operation S810, in response to receiving a target slice from the first processing unit, the second processing unit inputs a sample into a model to obtain output information, wherein the sample includes label information and the model is constructed according to the target slice.

[0139] In operation S820, the second processing unit determines a loss function value according to the output information and the label information.

[0140] In operation S830, the second processing unit adjusts the target slice of the model according to the loss function value to obtain a trained model.

[0141] In operation S840 , the second processing unit sends the adjusted target slice to the first processing unit.

[0142] The sample may include one of the following: a sample image, a sample text, and a sample audio. For example, in the case where the sample includes a sample image, the trained model determined by the task execution method provided according to the embodiment of the present disclosure may be used in the field of image processing. Alternatively, in the case where the sample includes a sample text, the trained model determined by the task execution method provided according to the embodiment of the present disclosure may be used in the field of text processing. Alternatively, in the case where the sample includes a sample audio, the trained model determined by the task execution method provided according to the embodiment of the present disclosure may be used in the field of speech processing.

[0143] According to an embodiment of the present disclosure, after receiving the target slice from the first processing unit, the second processing unit can construct a model locally in the second processing unit according to the target slice. By using the output information and label information to train the model, it is possible to achieve iterative optimization of the local model of the second processing unit, thereby improving the efficiency of task execution. In addition, it is conducive to the subsequent use of the trained model to process the data to be processed, realizing automatic processing of the data to be processed, thereby improving the efficiency of data processing.

[0144] Fig. 9 An example schematic diagram of a task execution process according to an embodiment of the present disclosure is schematically shown.

[0145] like Fig. 9 As shown, in 900, after receiving the target slice 901 from the first processing unit, the second processing unit can construct a model 902 to be trained based on the target slice 901. On this basis, the second processing unit can input a sample 903 into the model 902 to be trained to obtain output information 905. The sample 903 can include label information 904. The label information 904 can be used to characterize the true category corresponding to the sample 903. The output information 905 can be used to characterize the predicted category corresponding to the sample 903.

[0146] After obtaining the output information 905, the second processing unit can obtain the loss function value 907 based on the loss function 906, using the output information 905 and the label information 904. The second processing unit can adjust the model parameters of the model to be trained 902, that is, the target slice, according to the loss function value 907 until the predetermined condition is met. On this basis, the model obtained when the predetermined condition is met can be determined as a trained model. The predetermined condition may include at least one of the convergence of the loss function value and the training round reaching the maximum training round. After obtaining the trained model, the second processing unit can send the adjusted target slice 908 obtained when the predetermined condition is met to the first processing unit 909.

[0147] This is only an exemplary embodiment, but is not limited thereto, and may also include other task execution methods known in the art as long as the efficiency of task execution can be improved.

[0148] Fig.10 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0149] like Fig.10 As shown, the data processing method 1000 may include operations S1010 to S1020.

[0150] In operation S1010, the second processing unit obtains data to be processed.

[0151] In operation S1020, the second processing unit inputs the data to be processed into the trained model to obtain a data processing result.

[0152] The trained model is obtained using the task execution method 900. After obtaining the trained model, the second processing unit can obtain the data to be processed. The data to be processed may include at least one of the following: image data to be processed, text data to be processed, and audio data to be processed. After obtaining the data to be processed, the second processing unit can input the data to be processed into the trained model to obtain a data processing result.

[0153] According to the embodiments of the present disclosure, by acquiring the data to be processed and using the trained model to process the data to be processed to obtain the data processing result, it is possible to automatically analyze and process the data to be processed using the trained model, thereby improving the efficiency and accuracy of data processing.

[0154] This is only an exemplary embodiment, but is not limited thereto, and may also include other data processing methods known in the art as long as the efficiency of data processing can be improved.

[0155] Fig.11 The block diagram of a task processing device according to an embodiment of the present disclosure is schematically shown.

[0156] like Fig.11 As shown, the task processing device 1100 may include a first acquisition module 1110 , a first determination module 1120 , a conversion processing module 1130 and a first sending module 1140 .

[0157] The first acquisition module 1110 is used for, in response to detecting a conversion task, the first processing unit to acquire original model state information according to the model identifier indicated by the conversion task, wherein the conversion task includes an original parallel strategy identifier and a target parallel strategy identifier.

[0158] The first determination module 1120 is used for the first processing unit to determine the conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier.

[0159] The conversion processing module 1130 is used for the first processing unit to perform conversion processing on the original model state information based on the conversion strategy to obtain the target model state information.

[0160] The first sending module 1140 is used for the first processing unit to send the target model state information to multiple second processing units.

[0161] According to an embodiment of the present disclosure, the first determining module includes a first determining submodule and a second determining submodule.

[0162] The first determination submodule is used to determine a first conversion strategy according to the original parallel strategy identifier, wherein the first conversion strategy is used to convert the original model state information into the intermediate model state information.

[0163] The second determination submodule is used to determine a second conversion strategy according to the target parallel strategy identifier, wherein the second conversion strategy is used to convert the first intermediate model state information into the target model state information.

[0164] According to an embodiment of the present disclosure, the original model state information includes at least one original slice.

[0165] According to an embodiment of the present disclosure, in the case that the original parallel strategy corresponding to the original parallel strategy identifier includes the first parallel strategy, the first determining submodule includes a first determining unit.

[0166] The first determination unit is used to determine a first processing strategy and an aggregation strategy according to an original parallel strategy identifier, wherein the first processing strategy is used to perform a rearrangement operation on at least one original slice, and the aggregation strategy is used to perform an aggregation operation on at least one arranged original slice.

[0167] According to an embodiment of the present disclosure, in the case where the target parallel strategy corresponding to the target parallel strategy identifier includes the first parallel strategy, the second determining submodule includes a second determining unit.

[0168] The second determination unit is used to determine a segmentation strategy and a second processing strategy according to a target parallel strategy identifier, wherein the segmentation strategy is used to perform a segmentation operation on the intermediate model state information, and the second processing strategy is used to perform a rearrangement operation on at least one segmented target slice.

[0169] According to an embodiment of the present disclosure, the conversion processing module includes a first conversion processing sub-module and a second conversion processing sub-module.

[0170] The first conversion processing submodule is used to perform conversion processing on the original model state information based on the first conversion strategy to obtain the intermediate model state information.

[0171] The second conversion processing submodule is used to perform conversion processing on the intermediate model state information based on the second conversion strategy to obtain the target model state information.

[0172] According to an embodiment of the present disclosure, the first conversion strategy includes a first processing strategy and an aggregation strategy, and the original model state information includes at least one original slice.

[0173] According to an embodiment of the present disclosure, the first conversion processing submodule includes a first rearrangement unit and an aggregation unit.

[0174] The first rearrangement unit rearranges at least one original slice based on a first processing strategy to obtain at least one arranged original slice.

[0175] The aggregation unit is used to aggregate at least one arranged original slice based on an aggregation strategy to obtain intermediate model state information.

[0176] According to an embodiment of the present disclosure, each original slice has an information item identifier and an original slice identifier.

[0177] According to an embodiment of the present disclosure, the first rearrangement unit includes a determination subunit and a rearrangement subunit.

[0178] The determination subunit is used to determine at least one original slice to be arranged in at least one original slice according to the information item identifier of each original slice.

[0179] The rearrangement subunit is used to rearrange at least one original slice to be arranged according to the original slice identifier of each original slice to be arranged, so as to obtain at least one arranged original slice corresponding to the information item identifier.

[0180] According to an embodiment of the present disclosure, the aggregation unit includes an aggregation subunit.

[0181] The aggregation subunit is used to aggregate at least one arranged original slice according to the information item identifier to obtain the intermediate model state information corresponding to the information item identifier.

[0182] According to an embodiment of the present disclosure, the second conversion strategy includes a segmentation strategy and a second processing strategy.

[0183] According to an embodiment of the present disclosure, the second conversion processing submodule includes a segmentation unit and a second rearrangement unit.

[0184] The segmentation unit is used to segment the intermediate model state information based on the segmentation strategy to obtain at least one intermediate slice.

[0185] The second rearrangement unit rearranges at least one intermediate slice based on a second processing strategy to obtain target model state information.

[0186] According to an embodiment of the present disclosure, the segmentation strategy includes the number of second computing units.

[0187] According to an embodiment of the present disclosure, the segmentation unit includes a segmentation subunit.

[0188] The slicing subunit is used to slice the intermediate model state information corresponding to the information item identifier according to the number of second computing units for each information item identifier, so as to obtain at least one intermediate slice corresponding to the information item identifier.

[0189] According to an embodiment of the present disclosure, when the original parallel strategy corresponding to the original parallel strategy identifier includes the second parallel strategy and the target parallel strategy includes the second parallel strategy, the second determining submodule includes a third determining unit.

[0190] The third determining unit is used to determine a third processing strategy according to the original parallel processing identifier and the target parallel strategy identifier, wherein the third processing strategy is used to convert the original model state information into the target model state information.

[0191] According to an embodiment of the present disclosure, the original model state information includes at least one original slice.

[0192] According to an embodiment of the present disclosure, the conversion processing module includes a third rearrangement unit.

[0193] The third rearrangement unit is used to rearrange at least one original slice based on a third processing strategy to obtain target model state information.

[0194] According to an embodiment of the present disclosure, the task processing device further includes a setting module and a generating module.

[0195] The setting module is used to set an event listener, wherein the event listener is used to listen to a preset event corresponding to each of at least one candidate model identifier.

[0196] The generating module is used for generating a conversion task according to the model identifier by the first processing unit in response to the event listener being triggered.

[0197] According to an embodiment of the present disclosure, the target model state information includes at least one target slice.

[0198] According to an embodiment of the present disclosure, the first sending module includes a third determining submodule and a sending submodule.

[0199] The third determining submodule is used to determine a mapping relationship between a number of target slice groups of the second computing units and a plurality of second processing units, wherein each target slice group includes at least one target slice, and the mapping relationship includes a one-to-one correspondence between the target slice group and the second processing unit.

[0200] The sending submodule is used to synchronously send the number of target slice groups of the second computing units to the multiple second processing units according to the mapping relationship.

[0201] According to an embodiment of the present disclosure, the first acquisition module includes an acquisition submodule.

[0202] The acquisition submodule is used to acquire original model state information from a storage unit according to a model identifier in response to detecting a conversion task, wherein the storage unit stores at least one candidate model identifier and candidate original model state information of each model identifier.

[0203] According to an embodiment of the present disclosure, the task processing device further includes a first receiving module and an updating module.

[0204] The first receiving module is used for the first processing unit to receive the adjusted target slices from the plurality of second processing units.

[0205] The updating module is used for the first processing unit to update the original model state information stored in the storage unit using the adjusted target model state information, wherein the adjusted target model state information is determined according to a plurality of adjusted target slices.

[0206] According to an embodiment of the present disclosure, the first parallel strategy includes at least one of the following: a tensor parallel strategy and a first type of slice parallel strategy, and the second parallel strategy includes at least one of the following: a pipeline parallel strategy and a second type of slice parallel strategy.

[0207] According to an embodiment of the present disclosure, the original model state information includes original model parameters and original optimizer state parameters.

[0208] According to an embodiment of the present disclosure, the first processing unit includes a central processing unit in a master server, and the second processing unit includes at least one of the following in a slave server: a graphics processing unit, a central processing unit, and a neural network processing unit.

[0209] Fig.12 The block diagram of a task execution device according to an embodiment of the present disclosure is schematically shown.

[0210] like Fig.12As shown, the task execution device 1200 may include a first input module 1210 , a second determination module 1220 , an adjustment module 1230 and a second sending module 1240 .

[0211] The first input module 1210 is used for, in response to receiving the target slice from the first processing unit, the second processing unit inputs the sample into the model to obtain output information, wherein the sample includes label information and the model is constructed according to the target slice.

[0212] The second determination module 1220 is used for the second processing unit to determine the loss function value according to the output information and the label information.

[0213] The adjustment module 1230 is used for the second processing unit to adjust the target slice of the model according to the loss function value to obtain a trained model.

[0214] The second sending module 1240 is configured for the second processing unit to send the adjusted target slice to the first processing unit.

[0215] Fig.13 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0216] like Fig.13 As shown, the data processing device 1300 may include a second acquisition module 1310 and a second input module 1320 .

[0217] The second acquisition module 1310 is used for the second processing unit to acquire data to be processed.

[0218] The second input module 1320 is used for the second processing unit to input the data to be processed into the trained model to obtain the data processing result, wherein the trained model is obtained by using the task execution device.

[0219] Fig.14 A block diagram of an electronic device suitable for implementing a task processing method, a task execution method, and a data processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0220] like Fig.14As shown, the device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0221] A number of components in the device 1400 are connected to the I / O interface 1405, including: an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0222] The computing unit 1401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as the information display method. For example, in some embodiments, the information display method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the information display method described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to perform the information display method in any other appropriate manner (e.g., by means of firmware).

[0223] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0224] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0225] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0226] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0227] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0228] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0229] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0230] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A task processing method, comprising: In response to detecting a conversion task, the first processing unit acquires original model state information according to a model identifier indicated by the conversion task, wherein the conversion task includes an original parallel strategy identifier and a target parallel strategy identifier; The first processing unit determines a conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier; The first processing unit performs conversion processing on the original model state information based on the conversion strategy to obtain target model state information, wherein the target model state information includes at least one target slice; and The first processing unit sends the target model state information to a plurality of second processing units.

2. The method according to claim 1, wherein: The first processing unit determines the conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier, including: Determining a first conversion strategy according to the original parallel strategy identifier, wherein the first conversion strategy is used to convert the original model state information into intermediate model state information; and A second conversion strategy is determined according to the target parallel strategy identifier, wherein the second conversion strategy is used to convert the intermediate model state information into the target model state information.

3. The method according to claim 2, wherein: The original model state information includes at least one original slice; In the case where the original parallel strategy corresponding to the original parallel strategy identifier includes the first parallel strategy, Determining the first conversion strategy according to the original parallel strategy identifier includes: According to the original parallel strategy identifier, a first processing strategy and an aggregation strategy are determined, wherein the first processing strategy is used to perform a rearrangement operation on the at least one original slice, and the aggregation strategy is used to perform an aggregation operation on the at least one arranged original slice.

4. The method according to claim 2, wherein when the target parallel strategy corresponding to the target parallel strategy identifier includes a first parallel strategy, Determining the second conversion strategy according to the target parallel strategy identifier includes: According to the target parallel strategy identifier, a splitting strategy and a second processing strategy are determined, wherein the splitting strategy is used to perform a splitting operation on the intermediate model state information, and the second processing strategy is used to perform a rearrangement operation on at least one target slice after the splitting to obtain the target model state information.

5. The method according to claim 2, wherein: The first processing unit converts the original model state information based on the conversion strategy to obtain the target model state information including: Based on the first conversion strategy, converting the original model state information to obtain intermediate model state information; and Based on the second conversion strategy, the intermediate model state information is converted to obtain the target model state information.

6. The method according to claim 5, wherein: The first conversion strategy includes a first processing strategy and an aggregation strategy, and the original model state information includes at least one original slice; The converting process of the original model state information based on the first conversion strategy to obtain the intermediate model state information includes: Based on the first processing strategy, rearrange the at least one original slice to obtain at least one arranged original slice; as well as Based on the aggregation strategy, the arranged at least one original slice is aggregated to obtain the intermediate model state information.

7. The method according to claim 6, wherein: Each of the original slices has an information item identifier and an original slice identifier; The step of rearranging the at least one original slice based on the first processing strategy to obtain the at least one arranged original slice comprises: Determining at least one original slice to be arranged in the at least one original slice according to the information item identifier of each original slice; According to the original slice identifier of each of the original slices to be arranged, rearrange the at least one original slice to be arranged to obtain at least one arranged original slice corresponding to the information item identifier; and The step of aggregating the at least one arranged original slice based on the aggregation strategy to obtain the intermediate model state information includes: According to the information item identifier, the at least one arranged original slice is aggregated to obtain the intermediate model state information corresponding to the information item identifier.

8. The method according to claim 5, wherein: The second conversion strategy includes a segmentation strategy and a second processing strategy; The converting process of the intermediate model state information based on the second conversion strategy to obtain the target model state information includes: Based on the segmentation strategy, segment the intermediate model state information to obtain at least one intermediate slice; as well as Based on the second processing strategy, the at least one intermediate slice is rearranged to obtain the target model state information.

9. The method according to claim 8, wherein: The segmentation strategy includes the number of the second processing units; The segmenting of the intermediate model state information based on the segmentation strategy to obtain at least one intermediate slice comprises: For each information item identifier, The intermediate model state information corresponding to the information item identifier is segmented according to the number of the second processing units to obtain at least one intermediate slice corresponding to the information item identifier.

10. The method according to claim 2, wherein: In the case where the original parallel strategy corresponding to the original parallel strategy identifier includes the second parallel strategy and the target parallel strategy includes the second parallel strategy, Determining the second conversion strategy according to the target parallel strategy identifier includes: A third processing strategy is determined according to the original parallel strategy identifier and the target parallel strategy identifier, wherein the third processing strategy is used to convert the original model state information into the target model state information.

11. The method according to claim 10, wherein: The original model state information includes at least one original slice; The converting process of the original model state information based on the conversion strategy to obtain the target model state information includes: Based on the third processing strategy, the at least one original slice is rearranged to obtain the target model state information.

12. The method according to any one of claims 1 to 11, further comprising, before the first processing unit acquires the original model state information according to the model identifier indicated by the conversion task in response to detecting the conversion task: Set the event listener, where The event listener is used to listen to the preset events corresponding to at least one candidate model identifier; and In response to the event listener being triggered, the first processing unit generates the conversion task according to the model identifier.

13. The method according to any one of claims 1 to 11, wherein: The first processing unit sending the target model state information to the plurality of second processing units comprises: Determine a mapping relationship between a number of target slice groups of the second processing units and the plurality of second processing units, wherein each of the target slice groups includes at least one target slice, and the mapping relationship includes a one-to-one correspondence between the target slice groups and the second processing units; and According to the mapping relationship, the target slice groups of the number of the second processing units are synchronously sent to the multiple second processing units.

14. The method according to any one of claims 1 to 11, wherein: In response to detecting the conversion task, the first processing unit acquiring the original model state information according to the model identifier indicated by the conversion task includes: In response to detecting the conversion task, the original model state information is obtained from a storage unit according to the model identifier, wherein the storage unit stores at least one candidate model identifier and candidate original model state information of each candidate model identifier.

15. The method according to claim 14, further comprising, after the first processing unit sends the target model state information to a plurality of second processing units: The first processing unit receives the adjusted target slices from the plurality of second processing units; and The first processing unit updates the original model state information stored in the storage unit using the adjusted target model state information, wherein: The adjusted target model state information is determined based on a plurality of the adjusted target slices.

16. The method according to any one of claims 3, 4 or 10, wherein: The first parallel strategy includes at least one of the following: a tensor parallel strategy and a first type of slice parallel strategy, and the second parallel strategy includes at least one of the following: a pipeline parallel strategy and a second type of slice parallel strategy.

17. The method according to any one of claims 1 to 11, wherein: The original model state information includes original model parameters and original optimizer state parameters.

18. The method according to any one of claims 1 to 11, wherein: The first processing unit includes a central processing unit in a master server, and the second processing unit includes at least one of the following in a slave server: a graphics processing unit, a central processing unit, and a neural network processing unit.

19. A task execution method, comprising: In response to receiving the target slice from the first processing unit, the second processing unit inputs the sample into the model to obtain output information, wherein the sample includes label information, the model is constructed based on the target slice, and the target slice is obtained by using any one of the methods of claims 1 to 18; The second processing unit determines a loss function value according to the output information and the label information; The second processing unit adjusts the target slice of the model according to the loss function value to obtain a trained model; and The second processing unit sends the adjusted target slice to the first processing unit.

20. A data processing method, comprising: The second processing unit obtains the data to be processed; as well as The second processing unit inputs the data to be processed into a trained model to obtain a data processing result, wherein the trained model is obtained using the method described in claim 19.

21. A task processing device, comprising: A first acquisition module, configured to, in response to detecting a conversion task, obtain original model state information according to a model identifier indicated by the conversion task by a first processing unit, wherein the conversion task includes an original parallel strategy identifier and a target parallel strategy identifier; A first determination module, configured for the first processing unit to determine a conversion strategy according to the original parallel strategy identifier and the target parallel strategy identifier; a conversion processing module, configured for the first processing unit to perform conversion processing on the original model state information based on the conversion strategy to obtain target model state information, wherein the target model state information includes at least one target slice; and The first sending module is used for the first processing unit to send the target model state information to multiple second processing units.

22. The device according to claim 21, wherein The first determining module comprises: A first determining submodule is configured to determine a first conversion strategy according to the original parallel strategy identifier, wherein the first conversion strategy is configured to convert the original model state information into intermediate model state information; and The second determination submodule is configured to determine a second conversion strategy according to the target parallel strategy identifier, wherein the second conversion strategy is used to convert the intermediate model state information into the target model state information.

23. The device according to claim 22, wherein: The original model state information includes at least one original slice; In the case where the original parallel strategy corresponding to the original parallel strategy identifier includes the first parallel strategy, The first determining submodule includes: The first determination unit is used to determine a first processing strategy and an aggregation strategy according to the original parallel strategy identifier, wherein the first processing strategy is used to rearrange the at least one original slice, and the aggregation strategy is used to aggregate the at least one arranged original slice.

24. The apparatus according to claim 22, wherein when the target parallel strategy corresponding to the target parallel strategy identifier includes a first parallel strategy, The second determining submodule includes: The second determination unit is used to determine a segmentation strategy and a second processing strategy according to the target parallel strategy identifier, wherein the segmentation strategy is used to perform a segmentation operation on the intermediate model state information, and the second processing strategy is used to rearrange at least one segmented target slice to obtain the target model state information.

25. The device according to claim 22, wherein: The conversion processing module includes: A first conversion processing submodule, configured to perform conversion processing on the original model state information based on the first conversion strategy to obtain intermediate model state information; and The second conversion processing submodule is used to perform conversion processing on the intermediate model state information based on the second conversion strategy to obtain the target model state information.

26. The device according to claim 25, wherein The first conversion strategy includes a first processing strategy and an aggregation strategy, and the original model state information includes at least one original slice; The first conversion processing submodule includes: A first rearrangement unit, based on the first processing strategy, rearranges the at least one original slice to obtain at least one arranged original slice; as well as An aggregation unit is used to aggregate the at least one arranged original slice based on the aggregation strategy to obtain the intermediate model state information.

27. The device according to claim 26, wherein: Each of the original slices has an information item identifier and an original slice identifier; The first rearrangement unit comprises: a determination subunit, configured to determine at least one original slice to be arranged in the at least one original slice according to the information item identifier of each original slice; a rearrangement subunit, configured to rearrange the at least one original slice to be arranged according to the original slice identifier of each original slice to be arranged, so as to obtain at least one arranged original slice corresponding to the information item identifier; and The polymeric unit comprises: The aggregation subunit is used to aggregate the at least one arranged original slice according to the information item identifier to obtain the intermediate model state information corresponding to the information item identifier.

28. The device according to claim 25, wherein: The second conversion strategy includes a segmentation strategy and a second processing strategy; The second conversion processing submodule includes: A segmentation unit, configured to segment the intermediate model state information based on the segmentation strategy to obtain at least one intermediate slice; as well as The second rearrangement unit rearranges the at least one intermediate slice based on the second processing strategy to obtain the target model state information.

29. The device according to claim 28, wherein The segmentation strategy includes the number of the second processing units; The segmentation unit comprises: The slicing sub-unit is used to slice the intermediate model state information corresponding to each information item identifier according to the number of the second processing units to obtain at least one intermediate slice corresponding to the information item identifier.

30. The device according to claim 22, wherein: In the case where the original parallel strategy corresponding to the original parallel strategy identifier includes the second parallel strategy and the target parallel strategy includes the second parallel strategy, The second determining submodule includes: The third determining unit is used to determine a third processing strategy according to the original parallel strategy identifier and the target parallel strategy identifier, wherein the third processing strategy is used to convert the original model state information into the target model state information.

31. The device according to claim 30, wherein The original model state information includes at least one original slice; The conversion processing module includes: The third rearrangement unit is used to rearrange the at least one original slice based on the third processing strategy to obtain the target model state information.

32. The apparatus according to any one of claims 21 to 31, further comprising: a setting module, configured to set an event listener, wherein the event listener is configured to listen to a preset event corresponding to each of at least one candidate model identifier; and A generating module is used for generating the conversion task according to the model identifier by the first processing unit in response to the event listener being triggered.

33. The device according to any one of claims 21 to 31, wherein: The first sending module includes: a third determining submodule, configured to determine a mapping relationship between the number of target slice groups of the second processing units and the plurality of second processing units, wherein each of the target slice groups includes at least one of the target slices, and the mapping relationship includes a one-to-one correspondence between the target slice groups and the second processing units; and The sending submodule is used to synchronously send the target slice groups of the second processing units to the multiple second processing units according to the mapping relationship.

34. The device according to any one of claims 21 to 31, wherein: The first acquisition module includes: An acquisition submodule is used to acquire the original model state information from a storage unit in response to detecting the conversion task and according to the model identifier, wherein the storage unit stores at least one candidate model identifier and candidate original model state information of each candidate model identifier.

35. The apparatus of claim 34, further comprising: a first receiving module, configured for the first processing unit to receive the adjusted target slices from the plurality of second processing units; as well as An updating module is used for the first processing unit to update the original model state information stored in the storage unit using the adjusted target model state information, wherein the adjusted target model state information is determined based on a plurality of the adjusted target slices.

36. The device according to any one of claims 23, 24 or 30, wherein: The first parallel strategy includes at least one of the following: a tensor parallel strategy and a first type of slice parallel strategy, and the second parallel strategy includes at least one of the following: a pipeline parallel strategy and a second type of slice parallel strategy.

37. The device according to any one of claims 21 to 31, wherein: The original model state information includes original model parameters and original optimizer state parameters.

38. The device according to any one of claims 21 to 31, wherein: The first processing unit includes a central processing unit in a master server, and the second processing unit includes at least one of the following in a slave server: a graphics processing unit, a central processing unit, and a neural network processing unit.

39. A task execution device, comprising: a first input module, configured to, in response to receiving a target slice from the first processing unit, input a sample into a model by the second processing unit to obtain output information, wherein the sample includes label information, the model is constructed based on the target slice, and the target slice is obtained by using the apparatus according to any one of claims 21 to 38; A second determination module, configured for the second processing unit to determine a loss function value according to the output information and the label information; an adjustment module, configured for the second processing unit to adjust the target slice of the model according to the loss function value to obtain a trained model; and A second sending module is configured for the second processing unit to send the adjusted target slice to the first processing unit.

40. A data processing device, comprising: A second acquisition module, used for the second processing unit to acquire data to be processed; as well as The second input module is used for the second processing unit to input the data to be processed into the trained model to obtain a data processing result, wherein the trained model is obtained using the device described in claim 39.

41. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 20.

42. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 20.

43. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 20.

Images