Distributed data processing method, device, equipment, computer readable storage medium and computer program product

Abstract

The application provides a distributed data processing method, device, equipment, computer readable storage medium and computer program product. The method comprises: performing data synchronization on a plurality of devices based on the latest data of a first parameter group to which a first parameter to be executed belongs, wherein the first parameter is derived from a plurality of parameters, and the first parameter group is derived from a plurality of parameter groups; in response to completion of data synchronization for the first parameter group, performing first forward propagation of a corresponding first model based on the first parameter, and performing data synchronization on the plurality of devices based on the latest data of a second parameter group, wherein the second parameter group is a parameter group that is executed after the first parameter group in the first model. Through the application, the latest data of the second parameter group can be synchronized on the plurality of devices while performing the forward propagation of the corresponding first model based on the first parameter, parallel execution of the forward propagation and the data synchronization is realized, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a distributed data processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] The relevant technology employs a mode of synchronizing all parameters before initiating forward propagation: first, the latest data of all parameter groups in the model are synchronized across multiple devices, ensuring that the parameters on each device are completely consistent, before the forward propagation calculation of the model begins. Full parameter synchronization requires the transfer of a large amount of data, which is not only time-consuming, but also leaves resources used for forward propagation calculations idle during the synchronization process. Furthermore, parameter synchronization and forward propagation calculations are executed sequentially, resulting in low overall process efficiency, especially for very large-scale models, which can significantly increase training or inference time and lead to low resource utilization. Summary of the Invention

[0003] This application provides a distributed data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can synchronize the latest data of the second parameter group on multiple devices while performing forward propagation of the corresponding first model based on the first parameter, thereby achieving parallel execution of forward propagation and data synchronization and improving data processing efficiency.

[0004] The technical solution of this application embodiment is implemented as follows: This application embodiment provides a distributed data processing method, the method being applied to a distributed system, the distributed system including multiple devices, multiple parameters of the first model being divided into multiple parameter groups, and for each parameter group, the parameters of the parameter group being distributed to the multiple devices, the method including:

[0005] Based on the latest data of the first parameter group to which the first parameter to be executed belongs, data synchronization is performed on the multiple devices, wherein the first parameter originates from the multiple parameters, and the first parameter group originates from the multiple parameter groups;

[0006] In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameters, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is the parameter group in the first model that is executed after the first parameter group.

[0007] This application provides a distributed data processing device, which includes a first data synchronization module and a second data synchronization module. The processor is used to implement a distributed data processing method, which is applied to a distributed system. The distributed system includes multiple devices. Multiple parameters of the first model are divided into multiple parameter groups, and for each parameter group, the parameters of the parameter group are distributed to the multiple devices.

[0008] The first data synchronization module is used to perform data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, wherein the first parameter comes from the multiple parameters and the first parameter group comes from the multiple parameter groups;

[0009] The second data synchronization module is used to, in response to the completion of data synchronization for the first parameter group, perform a first forward propagation corresponding to the first model based on the first parameters, and perform data synchronization on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is the parameter group in the first model that is executed after the first parameter group.

[0010] In the above scheme, the device further includes a third data synchronization module, which is used to continue to perform data synchronization on the multiple devices based on the latest data of the third parameter group in response to the completion of data synchronization for the second parameter group, until all parameter groups are synchronized on the multiple devices, wherein the third parameter group is the parameter group whose execution order is after the second parameter group; and to perform a second forward propagation corresponding to the first model based on the second parameter group in response to the completion of the first forward propagation, until forward propagation is completed based on all parameter groups.

[0011] In the above scheme, the third data synchronization module is further configured to: determine the training loss of the first model based on the second forward propagation result corresponding to the second forward propagation; determine the first gradient corresponding to the second parameter group based on the training loss; update the second parameter group based on the first gradient, and determine the second gradient corresponding to the first parameter group based on the first gradient, so as to update the first parameter group based on the second gradient; in response to the completion of the second parameter group update, perform data synchronization on the multiple devices based on the updated second parameter group; and in response to the completion of the first parameter group update, perform data synchronization on the multiple devices based on the updated first parameter group.

[0012] In the above scheme, the first data synchronization module is further configured to perform the following process through each of the multiple devices after the parameters in the first parameter group are allocated to the multiple devices: synchronize the latest data of the parameters in the first parameter group allocated to the device to other devices, wherein the other devices are devices other than the device in the multiple devices.

[0013] In the above scheme, the first data synchronization module is further configured to, in response to the fact that the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to the multiple devices, perform data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs.

[0014] In the above scheme, the device further includes a forward propagation execution module, which is used to perform a third forward propagation corresponding to the first model based on the second parameter and the first forward propagation result corresponding to the first forward propagation, in order to obtain a third forward propagation result. The first parameter group also includes a second parameter.

[0015] In the above scheme, the device further includes a parameter group determination module, which is used to sort the multiple parameters based on the execution order of the multiple parameters to obtain a first parameter sequence; when the number of parameters in the first parameter sequence is not less than the first data volume, data is extracted from the first parameter sequence starting from the head, the extracted parameters that conform to the first data volume are grouped into a parameter group, and the extracted parameters are removed from the first parameter sequence; when the number of parameters in the first parameter sequence is less than the first data volume, the parameters in the first parameter sequence are grouped into a parameter group.

[0016] In the above scheme, the parameter group determination module is further configured to form a first initial parameter group from the parameters of the first parameter sequence; in response to the fact that the data volume of the first initial parameter group cannot be divided by the first threshold, to add virtual parameters to the first initial parameter group, so that the first initial parameter group after adding virtual parameters can be divided by the first threshold, and to use the first initial parameter group with the added virtual parameters as the parameter group.

[0017] In the above scheme, the parameter group determination module is further configured to form a second initial parameter group from the extracted parameters; in response to the fact that the amount of data in the second initial parameter group cannot be divided by the first threshold, virtual parameters are added to the second initial parameter group so that the second initial parameter group with the added virtual parameters can be divided by the first threshold, and the second initial parameter group with the added virtual parameters is used as the parameter group.

[0018] In the above scheme, the device further includes a parameter allocation module, which is used to evenly distribute the data of all parameters included in the parameter group to the multiple devices.

[0019] This application provides an electronic device, the electronic device comprising:

[0020] Memory is used to store executable instructions or computer programs.

[0021] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0022] This application provides a chip, the chip comprising:

[0023] Memory is used to store executable instructions or computer programs.

[0024] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0025] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in this application when executed by a processor.

[0026] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the distributed data processing method provided in this application.

[0027] The embodiments of this application have the following beneficial effects:

[0028] Based on the latest data of the first parameter group to which the first parameter to be executed belongs, data synchronization is performed on the multiple devices. The first parameter originates from the multiple parameters, and the first parameter group originates from the multiple parameter groups. By selectively synchronizing the data of the first parameter group on multiple devices, on-demand synchronization can be achieved, reducing data transmission volume and improving synchronization efficiency. In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameter, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, which is the parameter group in the first model whose execution priority follows that of the first parameter group. Immediately after the data synchronization of the first parameter group is completed, forward propagation based on the first parameter in the first parameter group is initiated. Simultaneously, data of the second parameter group is synchronized on multiple devices, allowing the forward propagation calculation and parameter group synchronization to run in parallel. This reduces waiting time and improves overall efficiency. In the parameter execution order of the first model, the second parameter group is prioritized after the first parameter group. The synchronization of the second parameter group only depends on the latest data of the second parameter group. Forward propagation based on the first parameter in the first parameter group does not depend on the second parameter group. Therefore, the parallel processing between the data synchronization of the second parameter group and the forward propagation based on the first parameter in the first parameter group does not affect the accuracy of the current calculation. This embodiment of the application achieves parallel execution of forward propagation and parameter group data synchronization, improving data processing efficiency. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the distributed data processing system architecture provided in the embodiments of this application;

[0030] Figure 2 This is a schematic diagram of the processor structure provided in an embodiment of this application;

[0031] Figure 3 This is a first flowchart illustrating the distributed data processing method provided in an embodiment of this application;

[0032] Figure 4 This is a second flowchart illustrating the distributed data processing method provided in the embodiments of this application;

[0033] Figure 5 This is a schematic diagram of the third process of the distributed data processing method provided in the embodiments of this application;

[0034] Figure 6 This is a schematic diagram of the flattened parameters provided in the embodiments of this application;

[0035] Figure 7 This is a schematic diagram of parameter binning provided in the embodiments of this application;

[0036] Figure 8This is a schematic diagram of device allocation provided in an embodiment of this application;

[0037] Figure 9 This is a first schematic diagram of parameter updating provided in an embodiment of this application;

[0038] Figure 10 This is a second schematic diagram of parameter updating provided in an embodiment of this application;

[0039] Figure 11 This is a third schematic diagram of parameter updating provided in an embodiment of this application;

[0040] Figure 12 This is a fourth schematic diagram of parameter updating provided in an embodiment of this application;

[0041] Figure 13 This is a timeline diagram of parameter updates provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0046] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0047] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] (1) Data Parallelism (DP): When a single device cannot accommodate the entire model, the model is divided into multiple sub-models, and the calculation of the corresponding sub-models is performed on different computing devices to solve the problem of insufficient capacity of a single device and achieve data parallelism. At this time, each device stores a portion of the model's parameters.

[0050] (2) All-Gather: This is a basic collective communication operation in distributed computing, used to efficiently aggregate data among multiple devices, ensuring that each device can eventually obtain all local data. The core logic is that each device participating in the communication has a copy of the local data. After the all-gather operation, all devices will collect the local data of all other devices and splice these data into a complete global dataset in a preset order.

[0051] (3) Full parameters: This is the set of all parameters involved in the calculation and training of the model, covering all parameters of each layer and component of the model, such as weights and biases. In distributed scenarios, consistency needs to be maintained across multiple devices. Traditional methods would wait for the full parameters, i.e., all parameters, to be synchronized before performing forward propagation calculations.

[0052] (4) Collective communication: a mode of data exchange between multiple devices in a distributed system. It achieves data synchronization between different devices through standardized operations such as broadcasting and reduction, and is used to ensure the efficiency and consistency of large-scale distributed computing.

[0053] The relevant technology employs a mode of synchronizing all parameters before initiating forward propagation: first, the latest data of all parameter groups in the model are synchronized across multiple devices, ensuring that the parameters on each device are completely consistent, before the forward propagation calculation of the model begins. Full parameter synchronization requires the transfer of a large amount of data, which is not only time-consuming, but also leaves resources used for forward propagation calculations idle during the synchronization process. Furthermore, parameter synchronization and forward propagation calculations are executed sequentially, resulting in low overall process efficiency, especially for very large-scale models, which can significantly increase training or inference time and lead to low resource utilization.

[0054] This application provides a distributed data processing method, apparatus, device, computer-readable storage medium, and computer program product. It enables simultaneous forward propagation of a corresponding first model based on first parameters on multiple devices, while synchronizing the latest data of a second parameter group. This achieves parallel execution of forward propagation and data synchronization, improving data processing efficiency. The following describes exemplary applications of the electronic devices provided in this application. These devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following describes exemplary applications when the device is implemented as a server.

[0055] See Figure 1 , Figure 1 This is a schematic diagram of the distributed data processing system architecture provided in the embodiments of this application. In order to support a distributed data processing application, in the distributed data processing system 100, the first device 400-1, the second device 400-2, the third device 400-3 and the fourth device 400-4 are all connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of both. The server 200 uploads data to the database 500.

[0056] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal device. For example, the first device 400-1, the second device 400-2, the third device 400-3, and the fourth device 400-4 all perform data synchronization using the distributed data processing method provided in the present application embodiments, and send the parameter group that has completed data synchronization to the server 200. The server 200 receives the parameter group that has completed data synchronization and performs forward propagation calculation. At the same time, the first device 400-1, the second device 400-2, the third device 400-3, and the fourth device 400-4 all perform data synchronization processing for the next parameter group using the distributed data processing method provided in the present application embodiments.

[0057] In some embodiments, the first device 400-1 stores a portion of the parameters of the first parameter group of the first model, i.e., the first parameter fragment, and the second device 400-2 stores another portion of the parameters of the first parameter group of the first model, i.e., the second parameter fragment. The union of the first parameter fragment stored on the first device 400-1 and the second parameter fragment stored on the second device 400-2 is the complete first parameter group. Data synchronization is initiated by the device storing the latest data fragment of the first parameter group. For example, when the first parameter fragment stored by the first device 400-1 is the latest data of the first parameter group, the first device 400-1 initiates data synchronization. That is, the first device 400-1 performs data synchronization on the multiple devices, such as the first device 400-1 and the second device 400-2, based on the latest data of the first parameter to be executed within the first parameter group. Specifically, the first device 400-1 sends its stored data to the second device 400-2 via aggregated communication, and the second device 400-2 sends its stored data back to the first device 400-1 via aggregated communication. Ultimately, both the first device 400-1 and the second device 400-2 store the data corresponding to the first device 400-1 and the data corresponding to the second device 400-2, which is the complete latest data of the first parameter group. The first parameter originates from the multiple parameters, and the first parameter group originates from the multiple parameter groups. In response to the completion of data synchronization for the first parameter group, the server 200 can obtain the latest complete data of the first parameter group and perform the first forward propagation corresponding to the first model based on the first parameters.

[0058] In some embodiments, in the parameter execution order of the first model, the second parameter group is the parameter group in the first model whose execution order follows the first parameter group. Data synchronization of the second parameter group only needs to rely on the latest data of the second parameter group. The first forward propagation based on the first parameter in the first parameter group does not need to rely on the second parameter group. Therefore, while performing the first forward propagation corresponding to the first model based on the first parameter, the data of the second parameter group can be synchronized. The third device 400-3 stores a portion of the parameters of the second parameter group of the first model, i.e., the third parameter fragment. The fourth device 400-4 stores another portion of the parameters of the second parameter group of the first model, i.e., the fourth parameter fragment. The union of the third parameter fragment stored on the third device 400-3 and the fourth parameter fragment stored on the fourth device 400-4 is the complete second parameter group. Data synchronization is initiated by the device storing the latest data of the second parameter group. For example, when the third parameter segment stored by the third device 400-3 is the latest data of the second parameter group, the third device 400-3 initiates data synchronization. That is, the third device 400-3 performs data synchronization on the multiple devices, such as the third device 400-3 and the fourth device 400-4, based on the latest data of the second parameter group. In other words, the third device 400-3 sends the stored data to the fourth device 400-4 through aggregated communication, and the fourth device 400-4 sends the stored data to the third device 400-3 through aggregated communication. Finally, both the third device 400-3 and the fourth device 400-4 store the data corresponding to the third device 400-3 and the data corresponding to the fourth device 400-4, which is the complete latest data of the second parameter group.

[0059] It should be noted that the first device 400-1 and the third device 400-3 can be the same device or different devices, and the second device 400-2 and the fourth device 400-4 can be the same device or different devices; there are no restrictions here.

[0060] Furthermore, the first parameter fragment stored on the first device 400-1 is updated by the central processing unit (CPU). That is, when the first parameter fragment needs updating, the first device 400-1 uploads the fragment to be updated to the CPU, which then updates it and sends the updated fragment back to the first device 400-1, thus completing the update of the first parameter fragment stored on the first device 400-1. The updating of the parameter fragments stored on the second device 400-2, the third device 400-3, and the fourth device 400-4 is the same as that of the first device 400-1, and will not be described further here.

[0061] From the perspective of internal system communication, the embodiments of this application effectively improve the latency and throughput issues caused by large data volumes. By storing parameters in fragments across different devices, the amount of data in a single communication between devices can be significantly reduced, thus decreasing data transmission latency. Aggregated communication enables parallel data exchange between devices, improving data throughput per unit time. Simultaneously, the parallel design of computation and communication allows the system to synchronize subsequent parameters while processing preceding parameter calculations, fully utilizing communication bandwidth and avoiding the problem of idle communication links in traditional serial modes. The mechanism of centralized parameter fragment updates by the central processing unit also reduces redundant communication across devices, further optimizing the efficiency of internal data flow and alleviating the pressure on throughput performance from large-scale data transmission.

[0062] The distributed data processing method provided in this application can be applied to image processing scenarios. From the full set of parameters of the image processing model (first model), such as convolutional kernel weights and normalization parameters, a first parameter responsible for initial feature extraction and its corresponding first parameter group are selected. For example, the complete parameter set of the first convolutional layer. After obtaining the latest data of the first parameter group, synchronization is performed across multiple devices, such as graphics processing unit (GPU) nodes and edge computing devices, via aggregated communication to ensure consistency of the initial processing parameters of the images stored on each device. Once the first parameter group synchronization is complete, each device performs a first forward propagation based on the parameters of the first parameter group, such as performing convolution operations on the image to extract primary features such as edges and textures. Simultaneously, a second parameter group is determined, whose execution order in the image processing model (first model) follows the first parameter group. For example, the parameters of the batch normalization layer after the convolutional layer and the next convolutional layer. The latest data of the second parameter group is obtained, and multi-device synchronization is completed, providing a consistent parameter basis for further image processing, such as normalization and advanced feature extraction. Through staged parameter group synchronization and forward propagation, the parallel efficiency of image processing is improved while ensuring the consistency of multi-device computation.

[0063] The distributed data processing method provided in this application can be applied to text processing scenarios. From the full set of parameters of the text processing model (first model), such as word embedding matrices, attention weights, and linear layer weights, a first parameter responsible for initial text feature encoding and its corresponding first parameter group are selected. For example, the complete parameter set of the word embedding layer, including the vector matrix corresponding to the vocabulary, is selected. After obtaining the latest data of the first parameter group, synchronization is performed across multiple devices using set communication to ensure that the initial text processing parameters stored on each device remain consistent. Once the first parameter group synchronization is complete, each device performs a first forward propagation based on this set of parameters. For example, word embedding operations are performed on the input text sequence, such as news sentences or comments, converting each word into a fixed-dimensional vector and extracting the basic semantic features of the text, such as the semantic information of the words themselves. Simultaneously, the second parameter group, which is executed after the first parameter group in the text processing model (first model), is determined. For example, the positional encoding layer parameters and attention layer parameters after the word embedding layer are selected. The latest data of the second parameter group is obtained, and multi-device synchronization is completed. This phased parameter group synchronization and forward propagation mechanism can ensure the computational consistency of multiple devices in the text processing process, and give full play to the advantages of multi-device parallelism, thereby improving the processing efficiency of tasks such as text classification, sentiment analysis, and machine translation.

[0064] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0065] See Figure 2 , Figure 2 This is a schematic diagram of the processor structure provided in an embodiment of this application. Figure 2 The illustrated electronic device 600 may be a first device 400-1, a second device 400-2, a third device 400-3, a fourth device 400-4, and a server 200. The electronic device 600 includes at least one processor 610, a memory 650, at least one network interface 620, and a user interface 630. The various components in the electronic device 600 are coupled together via a bus system 640. It is understood that the bus system 640 is used to implement communication between these components. In addition to a data bus, the bus system 640 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2The general labeled all buses as Bus System 640.

[0066] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] User interface 630 includes one or more output devices 631 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 630 also includes one or more input devices 632, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0068] The memory 650 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 650 may optionally include one or more storage devices physically located away from the processor 610.

[0069] The memory 650 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 650 described in this application embodiment is intended to include any suitable type of memory.

[0070] In some embodiments, memory 650 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0071] Operating system 651 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0072] The network communication module 652 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 620, exemplary network interfaces 620 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0073] Presentation module 653 enables the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 631 associated with user interface 630 (e.g., a display screen, a speaker, etc.).

[0074] The input processing module 654 is used to detect and translate one or more user inputs or interactions from one or more input devices 632.

[0075] In some embodiments, the processor provided in this application can be implemented in software. Figure 2 A distributed data processing device 655 stored in memory 650 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a first data synchronization module 6551, a second data synchronization module 6552, a third data synchronization module 6553, a forward propagation execution module 6554, a parameter group determination module 6555, and a parameter allocation module 6556. These modules are logically linked and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0076] In other embodiments, the processor provided in this application can be implemented in hardware. As an example, the processor provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the distributed data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can use one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0077] In some embodiments, the terminal or server can implement the distributed data processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0078] The distributed data processing method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0079] The following describes the distributed data processing method provided in the embodiments of this application. As mentioned above, the electronic device implementing the distributed data processing method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0080] See Figure 3 , Figure 3 This is a first flowchart illustrating the distributed data processing method provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained below. Figure 3 The main component of the steps is the equipment. It should be noted that the distributed data processing method provided in this application embodiment is applied to a distributed system, which includes multiple devices. Multiple parameters of the first model are divided into multiple parameter groups, and for each parameter group, the parameters of that parameter group are distributed to the multiple devices.

[0081] In some embodiments, dividing multiple parameters into multiple parameter groups can be achieved through the following technical solution: based on the execution order of the multiple parameters, sort the multiple parameters to obtain a first parameter sequence; when the number of parameters in the first parameter sequence is not less than a first data volume, extract data from the first parameter sequence starting from the head, group the extracted parameters that match the first data volume into a parameter group, and remove the extracted parameters from the first parameter sequence, wherein the data volume of the parameter group is the first data volume; when the number of parameters in the first parameter sequence is less than the first data volume, group the parameters of the first parameter sequence into a parameter group, wherein the data volume of the parameter group is the data volume of the first parameter sequence.

[0082] As an example, multiple parameters in a model have a specific execution order, determined by the dependencies between parameters and the model's computational logic. For instance, parameters in the hidden layers are executed first, while parameters in the output layers are executed later. Therefore, when extracting parameters to obtain parameter sets, it's crucial to ensure the execution order remains unchanged. Thus, before extraction, multiple parameters need to be sorted according to their execution order. The position of a parameter in the resulting first parameter sequence represents its execution order; parameters executed earlier are closer to the beginning of the first parameter sequence. For example, parameter... The position in the first parameter sequence is Then the parameter It is the first The parameters to be executed are: The first data volume is the maximum data volume that each parameter group can hold. When the number of parameters in the first parameter sequence is not less than the first data volume, it means that the first parameter sequence can be extracted into at least two parameter groups. In this case, data extraction needs to start from the head of the first parameter sequence, grouping the extracted parameters that are not greater than the first data volume into a parameter group, and removing the extracted parameters from the first parameter sequence so that data extraction can start again from the head of the first parameter sequence until there are no parameters left in the first parameter sequence, that is, all parameters in the first parameter sequence have corresponding parameter groups. When the number of parameters in the first parameter sequence is less than the first data volume, it means that the first parameter sequence with a number of parameters less than the first data volume corresponds to only one parameter group. Therefore, it is necessary to group the parameters of the first parameter sequence into a parameter group. In this case, the number of parameter groups is the data volume of the first parameter sequence. As an example, in the image processing model (first model), the parameter execution order is determined by the inter-layer dependencies of the network and the image computation logic. For example, convolutional layer parameters participate in feature extraction first, and pooling layer and fully connected layer parameters are executed sequentially. Therefore, this order must be maintained when extracting parameters into parameter groups. Before extraction, parameters are sorted according to execution order to obtain the first parameter sequence. The position of a parameter in the first parameter sequence represents its execution order; parameters that appear earlier participate in image processing first. The first data volume is the maximum capacity of each parameter group. If the number of parameters in the first parameter sequence is not less than the first data volume, parameters not exceeding the first data volume are extracted from the beginning to form a parameter group, and the extracted parameters are removed from the first parameter sequence. This process is repeated for the remaining parameters until no parameters remain in the first parameter sequence. If the number of parameters in the first parameter sequence is less than the first data volume, then the entire first parameter sequence constitutes the parameter group. The parameters in the first parameter sequence are flattened parameters. Flattening requires obtaining data from the parameters according to the first rule and concatenating the obtained data to obtain the flattened parameters. For example, for parameters... In this case, the first rule is to read line by line. The data obtained by reading line by line is [1,2] and [3,4]. Concatenating [1,2] and [3,4] together results in... That is, for parameters Flatten the data to obtain the flattened parameters. .

[0083] In some embodiments, the parameters of the first parameter sequence can be grouped into a parameter group by the following technical solution: the parameters of the first parameter sequence are grouped into a first initial parameter group; in response to the fact that the amount of data in the first initial parameter group cannot be divided by a first threshold, virtual parameters are added to the first initial parameter group so that the first initial parameter group with the added virtual parameters can be divided by the first threshold, and the first initial parameter group with the added virtual parameters is used as the parameter group.

[0084] As an example, since the parameters in each parameter group need to be evenly distributed across multiple devices to ensure that each device completes its task at a roughly consistent time, avoiding slowing down the overall progress due to some devices being overloaded or wasting resources due to some devices being idle, the number of devices is the first threshold. Therefore, it is necessary to ensure that the amount of data in the parameter group is divisible by the first threshold. The specific steps are as follows: Group the parameters of the first parameter sequence into a first initial parameter group; if the amount of data in the first initial parameter group is not divisible by the first threshold, then virtual parameters need to be added to the first initial parameter group so that the first initial parameter group with added virtual parameters is divisible by the first threshold, and the first initial parameter group with added virtual parameters is used as the parameter group. For example, if the first threshold is 3, the first initial parameter group is... The first initial parameter set has a data size of 5. Since 3 = 1 with a remainder of 2, the data size of the first initial parameter group is not divisible by the first threshold. Therefore, it is necessary to add virtual parameters to the first initial parameter group, such as... This ensures that the data size of the first initial parameter group after adding the dummy parameter is divisible by the first threshold. The minimum data size divisible by the first threshold is 6, therefore only one dummy parameter needs to be added to the first initial parameter group. That is, the first initial parameter group after adding the dummy parameter is... The data size of the first initial parameter group after adding virtual parameters is 6, which is divisible by the first threshold, i.e., the first initial parameter group with added virtual parameters. This is the parameter set. In image processing scenarios, the first parameter sequence, ordered by execution order, is first divided to obtain the first initial parameter set. The parameters in the first initial parameter set can include convolution kernel weights, bias terms, and other parameters related to image feature extraction. When the data size of the first initial parameter set cannot be divided by a first threshold (e.g., the first threshold is the size of the basic unit for parallel computing in image processing), virtual parameters need to be added to the first initial parameter set. These virtual parameters do not participate in the actual image feature calculation; they are only used to complete the data structure so that the completed first initial parameter set, i.e., the parameter set, can be divided by the first threshold. This ultimately forms a parameter set that can be directly used for parallel processing of image models, ensuring that multiple devices can achieve uniform distribution of data blocks and efficient parallel computing when processing image data.

[0085] In some embodiments, the extracted parameters can be grouped into a parameter group by the following technical solution: the extracted parameters are grouped into a second initial parameter group; in response to the fact that the amount of data in the second initial parameter group cannot be divided by a first threshold, virtual parameters are added to the second initial parameter group so that the second initial parameter group with the added virtual parameters can be divided by the first threshold, and the second initial parameter group with the added virtual parameters is used as the parameter group.

[0086] As an example, since the parameters in each parameter group need to be evenly distributed across multiple devices to ensure that each device completes its task at a roughly consistent time, avoiding slowing down the overall progress due to some devices being overloaded or wasting resources due to some devices being idle, the number of devices is the first threshold. Therefore, it is necessary to ensure that the amount of data in the parameter group is divisible by the first threshold. The specific steps are as follows: The extracted parameters are grouped into a second initial parameter group; if the amount of data in the second initial parameter group is not divisible by the first threshold, virtual parameters need to be added to the second initial parameter group so that the second initial parameter group with added virtual parameters is divisible by the first threshold, and the second initial parameter group with added virtual parameters is used as the parameter group. For example, if the first threshold is 3, the second initial parameter group is... The second initial parameter set has a data size of 5. Since 3 = 1 with a remainder of 2, the data size of the second initial parameter group is not divisible by the first threshold. Therefore, it is necessary to add virtual parameters to the second initial parameter group, such as... This ensures that the data size of the second initial parameter group after adding the dummy parameter is divisible by the first threshold. The minimum data size divisible by the first threshold is 6, therefore only one dummy parameter needs to be added to the second initial parameter group. That is, the second initial parameter group after adding the dummy parameter is... The data size of the second initial parameter group after adding dummy parameters is 6, which is divisible by the first threshold. That is, the second initial parameter group with added dummy parameters... This is the parameter set. In image processing scenarios, the first parameter sequence, ordered by execution order, is first divided to obtain a second initial parameter set. The parameters in the second initial parameter set can include convolution kernel weights, bias terms, and other parameters related to image feature extraction. When the data size of the second initial parameter set cannot be divided by the first threshold (e.g., the first threshold is the size of the basic unit for parallel computing in image processing), virtual parameters need to be added to the second initial parameter set. These virtual parameters do not participate in the actual image feature calculation; they are only used to complete the data structure so that the completed second initial parameter set, i.e., the parameter set, can be divided by the first threshold. This ultimately forms a parameter set that can be directly used for parallel processing of the image model, ensuring that multiple devices can achieve uniform distribution of data blocks and efficient parallel computing when processing image data.

[0087] Through the embodiments of this application, virtual parameters can be added to the initial parameter group, and the resulting parameter group is divisible by a first threshold. Since the embodiments of this application require the parameters in each parameter group to be evenly distributed across multiple devices, where the first threshold is the number of devices, adding virtual parameters to the initial parameter group ensures that the resulting parameter group is divisible by the first threshold, achieving the goal of even distribution across multiple devices. This ensures that the amount of parameter data stored on each device is basically consistent, resulting in roughly consistent task completion times for each device. This avoids slowing down the overall progress due to some devices being overloaded, or wasting resources due to some devices being idle.

[0088] Through the embodiments of this application, the parameter execution order is strictly followed, ensuring the coherence and correctness of the model logic. Furthermore, the strategy for obtaining the corresponding parameter group can be determined based on the relationship between the number of parameters in the first parameter sequence and the number of data in the first parameter group, so as to adapt to the first parameter sequence with different number of parameters and further improve the accuracy and rationality of determining the parameter group based on the first parameter sequence.

[0089] In some embodiments, each device is assigned a portion of the parameters of the first model. Assigning a portion of the parameters of the first model to each device can be achieved by the following technical solution: distributing the data of all parameters included in the parameter group evenly among the multiple devices.

[0090] By distributing the data of all parameters in each parameter group evenly across multiple devices through the embodiments of this application, it is possible to ensure that the amount of parameter data stored on each device is basically consistent, so that the time for the devices to complete the task is also basically consistent. This avoids the waste of resources caused by some devices being overloaded and slowing down the overall progress or some devices being idle when the devices are performing tasks.

[0091] In step 101, data synchronization is performed on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs.

[0092] As an example, the first parameter is derived from the plurality of parameters, and the first parameter group is derived from the plurality of parameter groups.

[0093] In some embodiments, Figure 3Step 101 shown can be implemented by the following technical solution: the parameters in the first parameter group are assigned to the plurality of devices. Each device performs the following process: synchronizing the latest data of the parameters assigned to that device in the first parameter group to other devices, wherein the other devices are devices other than the first device among the plurality of devices.

[0094] As an example, the parameters in the first parameter group are distributed across multiple devices. Specifically, the first device stores a portion of the parameters from the first parameter group of the first model, i.e., a first parameter fragment, while the second device stores the other portion of the parameters from the first parameter group of the first model, i.e., a second parameter fragment. The union of the first parameter fragment stored on the first device and the second parameter fragment stored on the second device constitutes the complete first parameter group. Data synchronization is initiated by the device storing the parameter fragment containing the latest data of the first parameter group. For example, if the first parameter fragment stored on the first device is the latest data of the first parameter group, the first device initiates data synchronization. That is, the first device performs data synchronization on multiple devices, such as the first device and the second device, based on the latest data of the first parameter group to which the first parameter to be executed belongs. In other words, the first device sends its stored parameters to the second device via aggregated communication, and the second device sends its stored parameters back to the first device via aggregated communication. Ultimately, both the first and second devices store the parameters corresponding to the first device and the parameters corresponding to the second device, i.e., the latest data of the complete first parameter group. The first parameter slice stored on the first device is updated by the central processing unit (CPU). That is, when the first parameter slice needs updating, the first device uploads the slice to be updated to the CPU, which then updates it and sends the updated slice back to the first device, completing the update of the first parameter slice stored on the first device. The update of the parameter slice stored on the second device is consistent with that of the first device, and will not be elaborated here. In image processing scenarios, when the first device holds the latest data of the first parameter slice allocated to it in a first parameter group (e.g., the parameters of a convolutional layer), the first device needs to synchronize the latest data of its stored first parameter slice to other devices participating in image processing, such as the second device. For example, after the first device responsible for extracting image edge features stores the latest convolutional kernel parameters, it sends these latest kernel parameters to the second device, which also participates in the feature calculation of that layer. This ensures that all devices use consistent parameters for image convolution, pooling, and other operations, avoiding feature extraction deviations caused by inconsistent parameters and guaranteeing computational consistency and result accuracy when multiple devices collaboratively process images.

[0095] Through the embodiments of this application, parameter synchronization between various devices enables multiple devices to quickly obtain the latest data of the complete first parameter group, avoiding deviations during forward propagation caused by devices having only partial parameters.

[0096] In some embodiments, Figure 3 Step 101 shown can also be implemented by the following technical solution: in response to the fact that the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to the multiple devices, data synchronization is performed on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs.

[0097] As an example, the condition for triggering data synchronization is that the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to multiple devices. That is, the data stored on multiple devices may have inconsistent versions or may not be the complete latest data of the first parameter group. In this case, data synchronization needs to be performed on multiple devices. Specifically, the first device stores a portion of the parameters of the first parameter group of the first model, i.e., the first parameter fragment, and the second device stores another portion of the parameters of the first parameter group of the first model, i.e., the second parameter fragment. The union of the first parameter fragment stored on the first device and the second parameter fragment stored on the second device constitutes the complete first parameter group. Data synchronization is initiated by the device storing the latest data of the parameter fragments in the first parameter group. For example, if the first device stores the latest data of the first parameter group, the first device initiates data synchronization. That is, the first device performs data synchronization on multiple devices, such as the first device and the second device, based on the latest data of the first parameter group to which the first parameter to be executed belongs. Specifically, the first device sends its stored parameters to the second device via aggregated communication, and the second device sends its stored parameters back to the first device via aggregated communication. Ultimately, both the first and second devices store the parameters corresponding to the first device and the parameters corresponding to the second device, representing the complete latest data of the first parameter group. The updates of the parameter fragments stored on the third and fourth devices are consistent with those of the first device and will not be elaborated here. In image processing scenarios, when the first device holds the latest data of the first parameter fragment allocated to it within the first parameter group, such as the parameters of a convolutional layer, the first device needs to synchronize the latest data of its stored first parameter fragments to other devices participating in image processing, such as the second device. For example, after the first device responsible for extracting image edge features stores the latest convolution kernel parameters, the first device will send these latest convolution kernel parameters to the second device that also participates in the feature calculation of the same layer. This ensures that all devices use consistent parameters to perform image convolution, pooling and other operations, avoids feature extraction deviations caused by inconsistent parameters, and ensures the consistency of calculation and the accuracy of results when multiple devices process images collaboratively.

[0098] This application's embodiments enable data synchronization to be triggered even when the latest data of the first parameter group to which the first parameter belongs has not been synchronized to multiple devices. This ensures that all devices possess completely consistent and complete latest data of the first parameter group before executing the task. This on-demand synchronization avoids redundant synchronization operations when the latest data is already synchronized, reducing unnecessary bandwidth usage, device computing power consumption, and time loss. Resources are concentrated on scenarios that truly require synchronization, improving overall processing efficiency. Furthermore, triggering the operation only when the latest data is not synchronized ensures that all devices possess completely consistent and complete latest data of the first parameter group before executing the task. This eliminates execution deviations caused by missing information, guaranteeing the accuracy of results and the stability of processes in multi-device collaborative tasks.

[0099] See also Figure 3 In step 102, in response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameters.

[0100] As an example, after the data synchronization of the first parameter group is completed, all devices have the latest complete data of the first parameter group. Therefore, the first parameter in the first parameter group is also the latest first parameter. At this time, the first forward propagation of the corresponding first model needs to be performed based on the first parameter. In image processing scenarios, when the first parameter group, such as the parameters of a convolutional layer or pooling layer, is synchronized across multiple devices, each device holds the latest complete and consistent parameters of the first parameter group. At this time, the system immediately performs the first forward propagation of the first model based on the first parameter group, for example, to extract features from the input image: by performing convolution operations with the image pixel matrix using the synchronized first parameter group, such as the convolution kernel parameters, basic features such as edges and textures are generated, or the feature map dimension is compressed through pooling operations. This process ensures that multiple devices can perform calculations based on completely consistent parameters, guaranteeing the consistency of feature extraction results.

[0101] In some embodiments, Figure 3 Following step 102, the following technical solution can also be implemented: the first parameter group further includes a second parameter. Based on the second parameter and the first forward propagation result corresponding to the first forward propagation, a third forward propagation corresponding to the first model is performed to obtain a third forward propagation result.

[0102] As an example, the first parameter group originates from multiple parameter groups, and these multiple parameter groups are derived from the sequence of first parameters determined by the execution order. Therefore, the parameters in the first parameter group have a definite execution order. Here, the execution order of the second parameters in the first parameter group follows the execution order of the first parameters in the first parameter group. Therefore, after performing the first forward propagation of the corresponding first model based on the first parameters to obtain the first forward propagation result, a third forward propagation of the corresponding first model needs to be performed based on the first forward propagation result and the second parameters to obtain the third forward propagation result. In image processing scenarios, the first parameter group includes not only the parameters involved in initial feature extraction but also the second parameters, such as high-level convolutional layer weights and activation function parameters. After completing the first forward propagation based on the first parameters, such as extracting basic features like image edges and textures, the system will use the second parameters and the basic feature map obtained from the first forward propagation to perform the third forward propagation. For example, by using the advanced convolutional kernel corresponding to the second parameter to perform deeper feature fusion and abstraction on the basic feature map, an advanced feature map containing information such as target contour and local semantics is generated, providing more discriminative feature support for subsequent tasks such as image classification and object detection. The final third forward propagation result can be directly used for image recognition decision-making or further feature processing.

[0103] In this embodiment, a basic first forward propagation is first performed using a first parameter, and then a third forward propagation is performed by combining the obtained first forward propagation result with a second parameter. This design reduces the parameter computation pressure of a single propagation, allows the second parameter to specifically supplement the feature gaps in the first forward propagation, reduces information redundancy and loss, and more accurately extracts deep data correlations, thereby improving the effectiveness of the third forward propagation result and enhancing the accuracy and stability of the model processing task.

[0104] In step 103, data synchronization is performed on the multiple devices based on the latest data from the second parameter group.

[0105] As an example, the second parameter group is the parameter group in the first model whose execution order follows that of the first parameter group. It should be noted that steps 102 and 103 are executed simultaneously. Data synchronization of the second parameter group only depends on the latest data of the second parameter group. The first forward propagation based on the first parameter in the first parameter group does not depend on the second parameter group. Therefore, while performing the first forward propagation corresponding to the first model based on the first parameter, the data of the second parameter group can be synchronized. The specific steps for performing data synchronization on multiple devices based on the latest data of the second parameter group are as follows: the third device stores a portion of the parameters of the second parameter group of the first model, i.e., the third parameter fragment; the fourth device stores another portion of the parameters of the second parameter group of the first model, i.e., the fourth parameter fragment. The union of the third parameter fragment stored on the third device and the fourth parameter fragment stored on the fourth device constitutes the complete second parameter group. Data synchronization is triggered when the latest data of the second parameter group has not been synchronized to the multiple devices. Data synchronization is initiated by the device storing the latest parameter fragment of the second parameter group. For example, if the third device stores the latest data of the second parameter group in its third parameter fragment, the third device initiates data synchronization. This means the third device performs data synchronization on multiple devices, such as the third and fourth devices, based on the latest data of the second parameter group. Specifically, the third device sends its stored data to the fourth device via aggregated communication, and the fourth device sends its stored data back to the third device via aggregated communication. Ultimately, both the third and fourth devices store the corresponding data for the third and fourth devices, representing the complete latest data of the second parameter group. Here, the third device and the first device in step 101 can be the same device or different devices. Similarly, the fourth device and the second device in step 101 can be the same device or different devices; there are no restrictions. In image processing scenarios, when the third device holds the latest data of the first parameter fragment allocated to it within the second parameter group, such as the parameters of a convolutional layer, the third device needs to synchronize its stored latest data of the third parameter fragment to other devices participating in image processing, such as the fourth device. For example, after the third device responsible for extracting image edge features stores the latest convolution kernel parameters, the third device will send these latest convolution kernel parameters to the fourth device that also participates in the feature calculation of the same layer. This ensures that all devices use consistent parameters to perform image convolution, pooling and other operations, avoids feature extraction deviations caused by inconsistent parameters, and ensures the consistency of calculation and the accuracy of results when multiple devices process images collaboratively.

[0106] From the perspective of internal system communication, the embodiments of this application effectively improve the latency and throughput problems caused by large data volumes. By storing parameters in fragments across different devices, the amount of data in a single communication between devices can be significantly reduced, thus reducing data transmission latency. The aggregated communication method enables parallel data exchange between devices, improving data throughput per unit time. Simultaneously, the parallel design of computation and communication allows the system to synchronize subsequent parameters while processing preceding parameter calculations, fully utilizing communication bandwidth and avoiding the problem of idle communication links in traditional serial modes. The mechanism of centralized parameter fragment updates by the central processing unit also reduces redundant communication across devices, further optimizing the efficiency of internal data flow and alleviating the pressure on throughput performance from large-scale data transmission.

[0107] In some embodiments, Figure 3 After step 103 shown, see Figure 4 You can also execute steps 104 to 105, as described below. Figure 4 Please provide an explanation.

[0108] In step 104, in response to the completion of data synchronization for the second parameter group, data synchronization continues to be performed on the multiple devices based on the latest data of the third parameter group until all parameter groups have been synchronized on the multiple devices.

[0109] In step 105, in response to the completion of the first forward propagation, a second forward propagation corresponding to the first model is performed based on the second parameter set until the forward propagation is completed based on all parameter sets.

[0110] As an example, the third parameter group is the parameter group whose execution priority follows the second parameter group. In this embodiment, data synchronization needs to be performed on multiple devices sequentially according to the execution priority of the parameter groups until all parameter groups are synchronized on multiple devices. Furthermore, after the first forward propagation based on the parameters in the first parameter group is completed, forward propagation of the corresponding first model needs to be performed sequentially based on each parameter group according to the execution priority of the parameter groups until all parameter groups have completed forward propagation. Taking the third parameter group and the second parameter group as examples, the third parameter group is the parameter group whose execution priority follows the second parameter group. Since the data synchronization of the third parameter group only depends on the latest data of the third parameter group and does not depend on the second parameter group, and the second forward propagation of the corresponding first model based on the second parameter group does not depend on the third parameter group, the data of the third parameter group can be synchronized simultaneously while the second forward propagation of the corresponding first model is performed based on the second parameter group. That is, in this embodiment, forward propagation and data synchronization can be performed simultaneously, and the execution priority of the parameter group targeted by forward propagation precedes the execution priority of the parameter group targeted by data synchronization. In image processing scenarios, once the second parameter group, such as pooling layer parameters, is synchronized across multiple devices, the third parameter group, such as the cross-device data synchronization of fully connected layer weights, is immediately initiated. This process continues until all parameter groups, including parameters for each convolutional layer and normalization layer, are synchronized consistently across multiple devices. After the first forward propagation, such as low-level edge feature extraction, is completed, the synchronized second parameter group is immediately used to perform a second forward propagation, such as mid-level texture feature fusion. This process proceeds sequentially according to the parameter group execution order until all parameter groups have participated in the forward propagation—ultimately completing the complete extraction from the original image to high-level semantic features such as target category and location information. This ensures efficient integration of computation and synchronization in the image processing workflow, improving the overall efficiency of model inference or training.

[0111] Through the embodiments of this application, the synchronization of parameter groups across multiple devices is advanced in batches according to execution order. Data synchronization of subsequent parameter groups, such as the third parameter group, is only performed after the data synchronization of the preceding parameter group, for example, the second parameter group, is completed. This ensures the consistency of parameters across multiple devices. Simultaneously, during model forward propagation, the parameter groups are executed sequentially according to their execution order, ensuring orderly and continuous forward propagation. This avoids conflicts during parameter group data synchronization and allows model computation to proceed in sequence, reducing redundant interactions.

[0112] In some embodiments, see Figure 5 Based on Figure 4 After obtaining the second forward propagation result in step 105, steps 106 to 111 can be executed. The following is a combination of... Figure 5 Please provide an explanation.

[0113] In step 106, the training loss of the first model is determined based on the second forward propagation result corresponding to the second forward propagation.

[0114] As an example, the second forward propagation result is a prediction, and the training loss of the first model can be determined by determining the prediction, i.e., the cross-entropy loss or mean square loss between the second forward propagation result and the true result.

[0115] In step 107, the first gradient corresponding to the second parameter set is determined based on the training loss.

[0116] In step 108, the second parameter set is updated based on the first gradient.

[0117] As an example, the backpropagation algorithm is used to calculate the partial derivatives of the second parameter set with respect to the training loss using the chain rule; this first gradient reflects the degree to which changes in the second parameter set affect the training loss. Then, in conjunction with the optimizer, the second parameter set is adjusted based on the magnitude and direction of the training loss, causing it to decrease. This process is repeated until the training loss converges to a relatively optimal value. The corresponding second parameter set at this point is the optimal second parameter set, and the second parameter set is then updated to the optimal set.

[0118] In step 109, based on the first gradient corresponding to the second parameter group, the second gradient corresponding to the first parameter group is determined, so as to update the first parameter group based on the second gradient.

[0119] As an example, since the second parameter group is executed after the first parameter group during forward propagation, the second gradient of the first parameter group depends on the first gradient of the second parameter group. Therefore, the second gradient of the first parameter group needs to be determined based on the first gradient corresponding to the second parameter group. Specifically, the first gradient is multiplied by the derivative of the connection relationship between the first and second parameter groups to calculate the partial derivative of the first parameter group with respect to the training loss, which is the second gradient. The second gradient reflects the degree of influence of changes in the first parameter group on the training loss. Then, in conjunction with the optimizer, the first parameter group is adjusted according to the magnitude and direction of the training loss to make the training loss decrease. This process is repeated until the training loss converges to a better value. At this point, the corresponding first parameter group is the optimal first parameter group, and the first parameter group is updated to the optimal first parameter group.

[0120] In step 110, in response to the completion of the second parameter group update, data synchronization is performed on the multiple devices based on the updated second parameter group.

[0121] In step 111, in response to the completion of the first parameter group update, data synchronization is performed on the multiple devices based on the updated first parameter group.

[0122] As an example, after the second parameter group is updated, data synchronization needs to be performed on multiple devices to ensure that the updated second parameter group is stored on all devices. Similarly, after the first parameter group is updated, data synchronization needs to be performed on multiple devices to ensure that the updated first parameter group is stored on all devices. The data synchronization steps can be found in [link to documentation]. Figure 3 Step 101, as shown, will not be repeated here. It should be noted that since the execution order of the first parameter group precedes that of the second parameter group during forward propagation, step 111 above needs to be executed before step 110 above. That is, data synchronization of the updated first parameter group across multiple devices needs to be performed first, and then the first model performs forward propagation based on the updated first parameter group while simultaneously performing data synchronization of the updated second parameter group across multiple devices. Forward propagation and data synchronization are performed simultaneously because the data synchronization of the second parameter group is only related to the updated second parameter group, and the forward propagation of the first parameter group is only related to the first parameter group after data synchronization. Therefore, forward propagation and data synchronization can be performed concurrently.

[0123] As an example, in an image processing scenario, the second forward propagation result can be the image classification prediction label, such as "cat" or "dog". By calculating the cross-entropy loss between the predicted result and the true image label, the model training loss is determined, and the prediction bias is measured. Based on the training loss, the second parameter set is derived in reverse, such as the first gradient of the fully connected layer parameters, reflecting the degree of influence of the layer parameters on the loss. Using the first gradient, the second parameter set is updated through algorithms such as gradient descent to reduce classification error. The first parameter set is derived from the first gradient of the second parameter set, such as the second gradient of the bottom convolutional layer, and then the parameters of the first parameter set, such as the convolution kernel, are updated to make the low-level feature extraction more closely match the true image features. After the second and first parameter sets are updated, the updated parameters are immediately synchronized across multiple devices to ensure that all devices use the same latest parameters in subsequent training, maintaining the stability and convergence efficiency of the image processing model training.

[0124] In this embodiment, the model training loss is first accurately calculated using the second forward propagation result. Then, the first gradient of the second parameter set is determined based on the training loss and updated. Simultaneously, the second gradient of the first parameter set is derived based on the first gradient to update the first parameter set. This ensures that the parameter set updates follow the order dependency of forward propagation. After the first and second parameter sets are updated, data synchronization is performed promptly, synchronizing the updated first and second parameter sets to multiple devices. This avoids interference from outdated parameter sets in multi-device collaborative computation and ensures that the parameter sets on each device remain consistent and up-to-date, effectively improving the stability of model training, parameter set synchronization efficiency, and final training effect in a multi-device environment.

[0125] In this embodiment, data synchronization is performed on multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs. The first parameter originates from the multiple parameters, and the first parameter group originates from the multiple parameter groups. By selectively synchronizing the data of the first parameter group on multiple devices, on-demand synchronization can be achieved, reducing data transmission volume and improving synchronization efficiency. In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameter. Then, based on the latest data of the second parameter group, data synchronization is performed on the multiple devices. The second parameter group is the parameter group in the first model whose execution priority follows that of the first parameter group. Immediately after the data synchronization of the first parameter group is completed, forward propagation based on the first parameter in the first parameter group is initiated. Simultaneously, data synchronization of the second parameter group is performed on multiple devices, allowing forward propagation calculation and parameter group synchronization to proceed in parallel, reducing waiting time and improving overall efficiency. In the parameter execution order of the first model, the second parameter group follows the first parameter group. Synchronization of the second parameter group only depends on the latest data of the second parameter group. Forward propagation based on the first parameter in the first parameter group does not depend on the second parameter group. Therefore, the parallel processing between data synchronization of the second parameter group and forward propagation based on the first parameter in the first parameter group does not affect the accuracy of the current calculation. This embodiment of the application achieves parallel execution of forward propagation and parameter group data synchronization, improving the efficiency of data processing.

[0126] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0127] In traditional data-parallel training, all processors need to store the complete model (first model) state, including model parameters, gradients, and optimizer state. This redundant storage causes memory usage to increase with the number of processors, severely limiting the trainable scale of large models. Distributed storage of the model state using sharding techniques can significantly reduce memory usage. The three levels of optimization strategies are as follows:

[0128] Level 1: Only the optimizer state is partitioned, and each device stores the complete parameters and gradients;

[0129] Second level: Piece optimizer state and gradient, with complete parameters stored for each device;

[0130] Level 3: The slice optimizer's state, gradient, and parameters minimize memory usage.

[0131] However, during distributed optimizer training, after parameter updates, the shard parameters of each graphics processing unit (GPU) need to be synchronized in the data-parallel (DP) domain through an all-gather operation. Since communication in the data-parallel domain typically requires communication across computing nodes, i.e., between different machines, this process introduces significant communication latency, becoming a bottleneck for training efficiency.

[0132] In mainstream industry solutions, some training frameworks can support full-collection masking, but only the first level mentioned above. Some frameworks can support splitting from the first to the third level, but cannot achieve full-collection masking. Therefore, the method provided in this application aims to solve the performance bottleneck problem caused by full-collection communication during parameter updates in distributed optimizers. This application supports communication masking during parameter updates in distributed optimizers and allows asynchronous parameter updates without waiting to acquire all gradients.

[0133] An optimizer typically contains multiple parameters, their corresponding gradients, and optimizer states. These parameters are flattened before being passed into the optimizer. Each parameter needs to correspond to multiple values; for example, parameter 1 has 1024 values. The flattening process involves obtaining the parameter data according to a first rule and concatenating the obtained parameter data to obtain the flattened parameters (the first parameter sequence). For example, for parameter... In this case, the first rule is to read line by line. The data obtained by reading line by line is [1,2] and [3,4]. Concatenating [1,2] and [3,4] together results in... That is, for parameters Flatten the data to obtain the flattened parameters. See also Figure 6 , Figure 6 This is a schematic diagram of the flattened parameters provided in the embodiments of this application.

[0134] The flattened parameters need to be bucketed (parameter groups). The bucket size is set, and when the number of parameters in a bucket reaches its upper limit (first data volume), a new bucket is created, and the remaining parameters are placed in the new bucket. If the sum of the number of parameters in a bucket is not divisible by the size of the data parallel domain (first threshold), the data in that bucket is padded, for example, with the value 0 (dummy parameter). After padding, the sum of the number of parameters in the bucket is an integer multiple of the size of the data parallel domain. See also... Figure 7 , Figure 7 This is a schematic diagram of parameter binning provided in the embodiments of this application.

[0135] See Figure 8 , Figure 8 This is a schematic diagram of device allocation provided in an embodiment of this application. The parameters in each bucket are evenly distributed across multiple devices corresponding to the data parallel domain. The parameters are downloaded to the central processing unit (CPU) for updating, and the updated parameters are sent from the CPU to the corresponding devices. The updated optimizer state is also saved by each device. Each bucket serves as a group of parameters for optimizer updates and is updated accordingly. Since parameter updates are element-wise operations, the partitioning and updating of parameters has no impact on accuracy. See also... Figure 8 Each bucket is evenly distributed among devices in the data parallel domain. For example, the majority of parameter 1 in bucket 1 is assigned to device 0, the remaining portion of parameter 1, all of parameter 2, and the initial portion of parameter 3 are assigned to device 1, and the majority of parameter 3 is assigned to device 2. This process is repeated for each bucket. When a specified bucket is reached, the parameter is downloaded to the central processing unit (CPU). The CPU then transmits the updated parameter to the corresponding device and updates the corresponding portion of the parameter. Each device is equal, and communication between devices is achieved through aggregated communication. Forward propagation, backward propagation, and parameter updates occur on each device. Data synchronization across devices is only performed during the next forward propagation after the parameter update. See also... Figure 9 , Figure 9 This is a first schematic diagram of parameter updating provided in an embodiment of this application. In related technologies, all parameters are usually collected and updated synchronously at once before forward propagation, which causes long-term communication blockage during computation.

[0136] See Figure 10 , Figure 10This is a second schematic diagram of parameter updating provided in an embodiment of this application. In this embodiment, after starting forward propagation, when each trainable parameter is reached, it is necessary to determine whether the data on all devices in the bucket to which that parameter belongs has been synchronized (data synchronization). If not, a full collection is invoked to synchronize the data on all devices in the bucket to which that parameter belongs. After synchronization is complete, forward propagation is performed again, and a communication stream is created simultaneously. The next bucket is synchronized within this communication stream. At this time, the communication stream and the propagation stream are performed synchronously, completing the masking process.

[0137] See Figure 10 When parameter 1 (the first parameter) is executed, since the data of each device in bucket 1 (the first parameter group) has not been synchronized at this time, a full collection is invoked to synchronize all devices in bucket 1. After synchronization is complete, parameter 1 is executed again. At the same time, a communication stream is created to synchronize the devices in bucket 2 (the second parameter group) in advance. When parameters 2 / 3 are executed, since parameters 2 / 3 have already been synchronized in bucket 1, forward propagation is executed directly. When parameter 4 is executed, since bucket 2 has been synchronized in advance, there is no need to wait, and forward propagation is executed directly. This process continues until the model forward propagation and parameter synchronization are completed.

[0138] Since parameter synchronization and updates are performed in groups of buckets, there is no need to wait for the entire backpropagation to complete before updating and synchronizing parameters. Once the parameters in a bucket have completed backpropagation, they can be downloaded to the central processing unit for early updates.

[0139] See Figure 11 , Figure 11 This is a third schematic diagram illustrating parameter updates provided in an embodiment of this application. In a conventional distributed training process, the typical procedure involves forward propagation, backpropagation, parameter updates on a single device, and synchronizing the updated parameters to other devices. A bucket contains multiple parameters. During training, forward propagation is performed sequentially for bucket 1 / 2, and after obtaining the loss, backpropagation for bucket 2 / 1 begins. Once the gradients of all parameters are obtained, the optimizer is invoked for updates. After a device's parameters are updated, a full collection is used to synchronize the parameters across all devices; that is, each device needs to know the data on other devices. Once synchronization is complete, the next round of forward propagation begins.

[0140] See Figure 12 , Figure 12This is a fourth schematic diagram of parameter updating provided in this application embodiment. During backpropagation, when the gradient calculation in bucket 2 is completed, the parameters are downloaded to the central processing unit for updating, while the backpropagation of bucket 1 and parameter updates continue. Once all devices in buckets 1 and 2 have completed their respective parameter updates, forward propagation begins. When bucket 1 begins the next round of forward propagation, a full collection is required for parameter synchronization before starting forward propagation in bucket 1. Simultaneously, a full collection is performed on the communication stream to synchronize parameters in bucket 2. See also... Figure 13 , Figure 13 This is a timeline diagram of parameter updates provided in the embodiments of this application. During backpropagation, the computation of bucket 1 on the device side and the parameter updates of bucket 2 on the central processing unit side can be masked; in the next round of forward propagation, the parameter updates of bucket 2 in the full collection of the communication flow can be masked by the forward propagation of bucket 1 on the device side.

[0141] It is understood that in the embodiments of this application, data such as parameters are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0142] The following description continues to illustrate the exemplary structure of the distributed data processing device 655 provided in the embodiments of this application as a software module. In some embodiments, see [link to relevant documentation]. Figure 2 The software modules stored in the distributed data processing device 655 of the memory 650 may include:

[0143] The first data synchronization module 6551 is used to perform data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, wherein the first parameter comes from the multiple parameters and the first parameter group comes from the multiple parameter groups.

[0144] The second data synchronization module 6552 is used to, in response to the completion of data synchronization for the first parameter group, perform a first forward propagation corresponding to the first model based on the first parameters, and perform data synchronization on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is the parameter group in the first model that is executed after the first parameter group.

[0145] In some embodiments, the apparatus further includes a third data synchronization module 6553, configured to, in response to the completion of data synchronization for the second parameter group, continue to perform data synchronization on the plurality of devices based on the latest data of the third parameter group until all parameter groups are synchronized on the plurality of devices, wherein the third parameter group is the parameter group whose execution priority follows the second parameter group; and in response to the completion of the first forward propagation, perform a second forward propagation corresponding to the first model based on the second parameter group until forward propagation is completed based on all parameter groups.

[0146] In some embodiments, the third data synchronization module 6553 is further configured to: determine the training loss of the first model based on the second forward propagation result corresponding to the second forward propagation; determine the first gradient corresponding to the second parameter group based on the training loss; update the second parameter group based on the first gradient, and determine the second gradient corresponding to the first parameter group based on the first gradient, so as to update the first parameter group based on the second gradient; in response to the completion of the second parameter group update, perform data synchronization on the multiple devices based on the updated second parameter group; and in response to the completion of the first parameter group update, perform data synchronization on the multiple devices based on the updated first parameter group.

[0147] In some embodiments, the first data synchronization module 6551 is further configured to perform the following process through each of the plurality of devices when the parameters in the first parameter group are assigned to the plurality of devices: synchronizing the latest data of the parameters in the first parameter group assigned to the device to other devices, wherein the other devices are devices other than the device in the plurality of devices.

[0148] In some embodiments, the first data synchronization module 6551 is further configured to perform data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, in response to the fact that the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to the multiple devices.

[0149] In some embodiments, the apparatus further includes a forward propagation execution module 6554, which is used to perform a third forward propagation corresponding to the first model based on the second parameter and the first forward propagation result corresponding to the first forward propagation, in order to obtain a third forward propagation result. The first parameter group further includes a second parameter.

[0150] In some embodiments, the apparatus further includes a parameter group determination module 6555, configured to sort the plurality of parameters based on the execution order of the plurality of parameters to obtain a first parameter sequence; when the number of parameters in the first parameter sequence is not less than a first data volume, data extraction is performed on the first parameter sequence starting from the head, the extracted parameters that conform to the first data volume are grouped into a parameter group, and the extracted parameters are removed from the first parameter sequence, wherein the data volume of the parameter group is the first data volume; when the number of parameters in the first parameter sequence is less than the first data volume, the parameters of the first parameter sequence are grouped into a parameter group, wherein the data volume of the parameter group is the data volume of the first parameter sequence.

[0151] In some embodiments, the parameter group determination module 6555 is further configured to form a first initial parameter group by combining the parameters of the first parameter sequence; in response to the fact that the amount of data in the first initial parameter group cannot be divided by a first threshold, to add virtual parameters to the first initial parameter group, such that the first initial parameter group after adding virtual parameters can be divided by the first threshold, and to use the first initial parameter group with the added virtual parameters as the parameter group.

[0152] In some embodiments, the parameter group determination module 6555 is further configured to group the extracted parameters into a second initial parameter group; in response to the data volume of the second initial parameter group not being divisible by a first threshold, to add virtual parameters to the second initial parameter group, such that the second initial parameter group with the added virtual parameters is divisible by the first threshold, and to use the second initial parameter group with the added virtual parameters as the parameter group. In some embodiments, the device further includes a parameter allocation module 6556, configured to perform the following processing for each parameter group: to evenly distribute the data of all parameters included in the parameter group to the plurality of devices.

[0153] This application provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the distributed data processing method described above in this application.

[0154] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the distributed data processing method provided in this application.

[0155] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0156] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0157] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, such as in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, computer-executable instructions may be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0158] In summary, this application embodiment performs data synchronization on multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs. The first parameter originates from the multiple parameters, and the first parameter group originates from the multiple parameter groups. By selectively synchronizing the data of the first parameter group on multiple devices, on-demand synchronization can be achieved, reducing data transmission volume and improving synchronization efficiency. In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameter. Then, based on the latest data of the second parameter group, data synchronization is performed on the multiple devices. The second parameter group is the parameter group in the first model whose execution order follows the first parameter group. Immediately after the data synchronization of the first parameter group is completed, forward propagation based on the first parameter in the first parameter group is initiated. Simultaneously, data synchronization of the second parameter group is performed on multiple devices, allowing forward propagation calculation and parameter group synchronization to run in parallel, reducing waiting time and improving overall efficiency. In the parameter execution order of the first model, the second parameter group follows the first parameter group. Synchronization of the second parameter group only depends on the latest data of the second parameter group. Forward propagation based on the first parameter in the first parameter group does not depend on the second parameter group. Therefore, the parallel processing between data synchronization of the second parameter group and forward propagation based on the first parameter in the first parameter group does not affect the accuracy of the current calculation. This embodiment of the application achieves parallel execution of forward propagation and parameter group data synchronization, improving the efficiency of data processing.

[0159] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A distributed data processing method, characterized in that, The method is applied to a distributed system comprising multiple devices. Multiple parameters of a first model are divided into multiple parameter groups, and for each parameter group, the parameters of that parameter group are assigned to the multiple devices. The method includes: Based on the latest data of the first parameter group to which the first parameter to be executed belongs, data synchronization is performed on the multiple devices, wherein the first parameter originates from the multiple parameters, and the first parameter group originates from the multiple parameter groups; In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameters, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is the parameter group in the first model that is executed after the first parameter group.

2. The method according to claim 1, characterized in that, The method further includes: In response to the completion of data synchronization for the second parameter group, data synchronization continues to be performed on the multiple devices based on the latest data of the third parameter group until all parameter groups are synchronized on the multiple devices, wherein the third parameter group is the parameter group whose execution priority is after the second parameter group; In response to the completion of the first forward propagation, a second forward propagation corresponding to the first model is performed based on the second parameter set until the forward propagation is completed based on all parameter sets.

3. The method according to claim 2, characterized in that, The method further includes: Based on the second forward propagation result corresponding to the second forward propagation, the training loss of the first model is determined; The first gradient corresponding to the second parameter set is determined based on the training loss. The second parameter group is updated based on the first gradient, and the second gradient corresponding to the first parameter group is determined based on the first gradient corresponding to the second parameter group, so as to update the first parameter group based on the second gradient. In response to the completion of the second parameter group update, data synchronization is performed on the multiple devices based on the updated second parameter group; In response to the completion of the first parameter group update, data synchronization is performed on the multiple devices based on the updated first parameter group.

4. The method according to claim 1, characterized in that, The process of performing data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs includes: Each of the devices performs the following process: synchronizing the latest data of the parameters assigned to the device in the first parameter group to other devices, wherein the other devices are devices other than the device itself among the plurality of devices.

5. The method according to claim 1, characterized in that, The process of performing data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs includes: In response to the fact that the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to the plurality of devices, data synchronization is performed on the plurality of devices based on the latest data of the first parameter group to which the first parameter to be executed belongs.

6. The method according to claim 1, characterized in that, The first parameter group further includes a second parameter. After performing a first forward propagation corresponding to the first model based on the first parameter, the method further includes: Based on the second parameter and the first forward propagation result corresponding to the first forward propagation, a third forward propagation corresponding to the first model is performed to obtain a third forward propagation result.

7. The method according to claim 1, characterized in that, The process of dividing the multiple parameters into multiple parameter groups includes the following steps: Based on the execution order of the multiple parameters, the multiple parameters are sorted to obtain the first parameter sequence; When the number of parameters in the first parameter sequence is not less than the first data volume, data extraction is performed on the first parameter sequence starting from the head, the extracted parameters that conform to the first data volume are grouped into a parameter group, and the extracted parameters are removed from the first parameter sequence. When the number of parameters in the first parameter sequence is less than the number of data, the parameters in the first parameter sequence are grouped into a parameter group.

8. The method according to claim 7, characterized in that, The step of assembling the parameters of the first parameter sequence into a parameter group includes: The parameters of the first parameter sequence are grouped into a first initial parameter group; In response to the fact that the amount of data in the first initial parameter group cannot be divided by the first threshold, a virtual parameter is added to the first initial parameter group so that the first initial parameter group after adding the virtual parameter can be divided by the first threshold, and the first initial parameter group with the added virtual parameter is used as the parameter group. The step of grouping the extracted parameters that conform to the first data volume into a parameter group includes: The extracted parameters are combined into a second initial parameter set; In response to the fact that the amount of data in the second initial parameter group is not divisible by the first threshold, a virtual parameter is added to the second initial parameter group so that the second initial parameter group with the added virtual parameter is divisible by the first threshold, and the second initial parameter group with the added virtual parameter is used as the parameter group.

9. The method according to claim 1, characterized in that, The parameters of the parameter group are assigned to the multiple devices, including the following steps: The data of all parameters included in the parameter group are evenly distributed among the multiple devices.

10. A processor, characterized in that, The processor is used to execute the method according to any one of claims 1 to 9.

11. A chip, characterized in that, The chip includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 9.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 9.

14. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 9.

Images