Distributed data processing method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides a distributed data processing method, device and equipment, a computer readable storage medium and a computer program product. The method comprises the steps that data synchronization is executed on a plurality of devices based on latest data of a first parameter group to which a first parameter to be executed belongs, the first parameter is derived from a plurality of parameters, and the first parameter group is derived from a plurality of parameter groups; in response to completion of data synchronization for a first parameter set, a first forward propagation corresponding to the first model is performed based on the first parameter, and data synchronization is performed on the plurality of devices based on latest data of a second parameter set, where the second parameter set is a parameter set executed in the first model following the first parameter set. According to the method and the device, the latest data of the second parameter group can be synchronized on the plurality of devices while the forward propagation corresponding to the first model is executed based on the first parameter, so that the parallel execution of the forward propagation and the data synchronization is realized, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a distributed data processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] Related technologies use a model where full parameter synchronization precedes forward propagation: the latest data for all parameter groups in the model is first synchronized across multiple devices, ensuring that the parameters of each device are completely consistent before the model's forward propagation calculations begin. Full parameter synchronization requires the transmission of large amounts of data, which is not only time-consuming, but also leaves the resources used for forward propagation calculations idle while all parameters are synchronized. Parameter synchronization and forward propagation calculations are executed serially, making the overall process inefficient. This can significantly increase the time required for training or inference, and lead to low resource utilization, especially for very large models. Summary of the Invention

[0003] The embodiments of the present application provide a distributed data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can synchronize the latest data of the second parameter group on multiple devices while executing forward propagation corresponding to the first model based on the first parameter, thereby realizing parallel execution of forward propagation and data synchronization and improving data processing efficiency.

[0004] The technical solution of the embodiment of the present application is implemented as follows: The embodiment of the present application provides a distributed data processing method, the method being applied to a distributed system, the distributed system including multiple devices, multiple parameters of the first model being divided into multiple parameter groups, and for each parameter group, the parameters of the parameter group being divided among the multiple devices, the method comprising: performing data synchronization on the multiple devices based on the latest data of a first parameter group to which a first parameter to be executed belongs, wherein the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups; In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameters, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is a parameter group whose execution order in the first model is after the first parameter group.

[0005] An embodiment of the present application provides a distributed data processing device, the device including a first data synchronization module and a second data synchronization module, the processor being configured to implement a distributed data processing method, the method being applied to a distributed system, the distributed system including a plurality of devices, a plurality of parameters of a first model being divided into a plurality of parameter groups, and for each parameter group, the parameters of the parameter group being divided among the plurality of devices; The first data synchronization module is configured to perform data synchronization on the multiple devices based on the latest data of a first parameter group to which a first parameter to be executed belongs, wherein the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups; The second data synchronization module is used to perform a first forward propagation corresponding to the first model based on the first parameters in response to the completion of data synchronization for the first parameter group, and to perform data synchronization on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is a parameter group whose execution order in the first model is after the first parameter group.

[0006] In the above scheme, the device also includes a third data synchronization module, which is used to, in response to the completion of data synchronization for the second parameter group, continue to perform data synchronization on the multiple devices based on the latest data of the third parameter group until all parameter groups are synchronized on the multiple devices, wherein the third parameter group is the parameter group whose execution order is after the second parameter group; in response to the completion of the first forward propagation, perform the second forward propagation corresponding to the first model based on the second parameter group until the forward propagation is completed based on all parameter groups.

[0007] In the above scheme, the third data synchronization module is also used to determine the training loss of the first model based on the second forward propagation result corresponding to the second forward propagation; determine the first gradient corresponding to the second parameter group based on the training loss; update the second parameter group based on the first gradient, and determine the second gradient corresponding to the first parameter group based on the first gradient corresponding to the second parameter group, so as to update the first parameter group based on the second gradient; in response to the completion of the update of the second parameter group, perform data synchronization on the multiple devices based on the updated second parameter group; in response to the completion of the update of the first parameter group, perform data synchronization on the multiple devices based on the updated first parameter group.

[0008] In the above scheme, the first data synchronization module is also used to allocate the parameters in the first parameter group to the multiple devices, and perform the following processing through each of the devices: synchronize the latest data of the parameters in the first parameter group allocated to the device to other devices, wherein the other devices are devices other than the device among the multiple devices.

[0009] In the above scheme, the first data synchronization module is also used to perform data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, in response to the latest data of the first parameter group to which the first parameter to be executed belongs not being synchronized to the multiple devices.

[0010] In the above scheme, the device also includes a forward propagation execution module, which is used for executing a third forward propagation corresponding to the first model based on the second parameter and the first forward propagation result corresponding to the first forward propagation when the first parameter group also includes a second parameter to obtain a third forward propagation result.

[0011] In the above scheme, the device also includes a parameter group determination module, which is used to sort the multiple parameters based on the execution order of the multiple parameters to obtain a first parameter sequence; when the parameter quantity of the first parameter sequence is not less than the first data quantity, the first parameter sequence is extracted from the head, the extracted parameters that meet the first data quantity are grouped into a parameter group, and the extracted parameters are removed from the first parameter sequence; when the parameter quantity of the first parameter sequence is less than the first data quantity, the parameters of the first parameter sequence are grouped into a parameter group.

[0012] In the above scheme, the parameter group determination module is also used to group the parameters of the first parameter sequence into a first initial parameter group; in response to the data volume of the first initial parameter group not being divisible by a first threshold value, add a virtual parameter to the first initial parameter group so that the first initial parameter group after adding the virtual parameter can be divisible by the first threshold value, and the first initial parameter group to which the virtual parameter is added is used as the parameter group.

[0013] In the above scheme, the parameter group determination module is also used to form the extracted parameters into a second initial parameter group; in response to the data volume of the second initial parameter group not being divisible by the first threshold, a virtual parameter is added to the second initial parameter group so that the second initial parameter group after the virtual parameter is added can be divisible by the first threshold, and the second initial parameter group to which the virtual parameter is added is used as the parameter group.

[0014] In the above solution, the apparatus further includes a parameter distribution module, configured to evenly distribute data of all parameters included in the parameter group to the multiple devices.

[0015] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions or computer programs; The processor is used to implement the method provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the memory.

[0016] An embodiment of the present application provides a chip, comprising: a memory for storing computer-executable instructions or computer programs; The processor is used to implement the method provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in the embodiment of the present application when executed by a processor.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the distributed data processing method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects: Based on the latest data of a first parameter group to which a first parameter to be executed belongs, data synchronization is performed on the multiple devices, the first parameter being derived from the multiple parameters, and the first parameter group being derived from the multiple parameter groups; by specifically synchronizing the data of the first parameter group on the multiple devices, on-demand synchronization can be achieved, the amount of data transmitted can be reduced, and synchronization efficiency can be improved. In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameter, and data synchronization is performed on the multiple devices based on the latest data of a second parameter group, the second parameter group being a parameter group whose execution order in the first model is after the first parameter group. After the data synchronization of the first parameter group is completed, the forward propagation based on the first parameter in the first parameter group is immediately started, and the data of the second parameter group is synchronized on multiple devices at the same time, so that the forward propagation calculation and the parameter group synchronization are parallel, which reduces waiting time and improves overall efficiency. In the parameter execution order of the first model, the second parameter group is ranked after the first parameter group. The synchronization of the second parameter group only depends on the latest data of the second parameter group. The forward propagation based on the first parameter in the first parameter group does not need to rely on the second parameter group. Therefore, the parallel processing between the data synchronization of the second parameter group and the forward propagation based on the first parameter in the first parameter group does not affect the accuracy of the current calculation. The embodiment of the present application realizes the parallel execution of forward propagation and parameter group data synchronization, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the distributed data processing system architecture provided by an embodiment of the present application; Figure 2 is a schematic diagram of the structure of the processor provided in an embodiment of the present application; Figure 3 This is a first flow chart of the distributed data processing method provided in an embodiment of the present application; Figure 4 This is a second flow chart of the distributed data processing method provided in an embodiment of the present application; Figure 5 This is a third flow chart of the distributed data processing method provided in an embodiment of the present application; Figure 6 is a schematic diagram of parameters after flattening provided in an embodiment of the present application; Figure 7 This is a schematic diagram of parameter bucketing provided in an embodiment of the present application; Figure 8 This is a schematic diagram of device allocation provided by an embodiment of the present application; Figure 9 This is a first schematic diagram of parameter updating provided by an embodiment of the present application; Figure 10This is a second schematic diagram of parameter updating provided by an embodiment of the present application; Figure 11 This is a third schematic diagram of parameter updating provided by an embodiment of the present application; Figure 12 This is a fourth schematic diagram of parameter updating provided by an embodiment of the present application; Figure 13 This is a timeline diagram of parameter updates provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0025] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0026] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0027] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0028] (1) Data Parallelism (DP): When a single device cannot accommodate the entire model, the model is divided into multiple sub-models, and the calculation of the sub-models corresponding to the device is performed on different computing devices to solve the problem of insufficient capacity of a single device, thus achieving data parallelism. At this time, each device stores part of the model parameters.

[0029] (2) All-Gather: This is a fundamental collective communication operation in distributed computing, used to efficiently aggregate data across multiple devices, ensuring that each device ultimately has access to all local data. The core logic is that each device participating in the communication has a copy of local data. After the All-Gather operation, all devices collect the local data of all other devices and stitch these data together into a complete global data set in a pre-set order.

[0030] (3) Full parameters: This is the set of all parameters involved in the calculation and training of the model, covering all parameters of each layer and component of the model, such as weights and biases. In a distributed scenario, consistency must be maintained across multiple devices. The traditional method is to wait until all parameters are synchronized before performing forward propagation calculations.

[0031] (4) Collective communication: A mode of collaborative data exchange between multiple devices in a distributed system. Data synchronization between different devices is achieved through standardized operations such as broadcasting and reduction, which is used to ensure the efficiency and consistency of large-scale distributed computing.

[0032] Related technologies use a model where full parameter synchronization precedes forward propagation: the latest data for all parameter groups in the model is first synchronized across multiple devices, ensuring that the parameters of each device are completely consistent before the model's forward propagation calculations begin. Full parameter synchronization requires the transmission of large amounts of data, which is not only time-consuming, but also leaves the resources used for forward propagation calculations idle while all parameters are synchronized. Parameter synchronization and forward propagation calculations are executed serially, making the overall process inefficient. This can significantly increase the time required for training or inference, and lead to low resource utilization, especially for very large models.

[0033] The embodiments of the present application provide a distributed data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can synchronize the latest data of the second parameter group on multiple devices while executing the forward propagation corresponding to the first model based on the first parameter, thereby realizing the parallel execution of forward propagation and data synchronization and improving data processing efficiency. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The device provided by the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and car terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0034] See also Figure 1 , Figure 1 This is a structural diagram of the distributed data processing system architecture provided in an embodiment of the present application. In order to support a distributed data processing application, in the distributed data processing system 100, the first device 400-1, the second device 400-2, the third device 400-3 and the fourth device 400-4 are all connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The server 200 uploads data to the database 500.

[0035] In some embodiments, the embodiments of the present application can be implemented collaboratively by a server and a terminal device. For example, the first device 400-1, the second device 400-2, the third device 400-3, and the fourth device 400-4 all perform data synchronization using the distributed data processing method provided in the embodiments of the present application, and send the parameter group for which data synchronization has been completed to the server 200. The server 200 receives the parameter group for which data synchronization has been completed and performs forward propagation calculation. At the same time, the first device 400-1, the second device 400-2, the third device 400-3, and the fourth device 400-4 all perform data synchronization processing for the next parameter group using the distributed data processing method provided in the embodiments of the present application.

[0036] In some embodiments, first device 400-1 stores a portion of the parameters of the first parameter group of the first model, namely, a first parameter slice, and second device 400-2 stores another portion of the parameters of the first parameter group of the first model, namely, a second parameter slice. The union of the first parameter slice stored on first device 400-1 and the second parameter slice stored on second device 400-2 constitutes the complete first parameter group. Data synchronization is initiated by the device storing the parameter slice of the latest data of the first parameter group. For example, when the first parameter slice stored in the first device 400-1 is the latest data of the first parameter group, the data synchronization is initiated by the first device 400-1. That is, the first device 400-1 performs data synchronization on the multiple devices, such as the first device 400-1 and the second device 400-2, based on the latest data of the first parameter group to which the first parameter to be executed belongs. That is, the first device 400-1 sends the stored data to the second device 400-2 through collective communication, and the second device 400-2 sends the stored data to the first device 400-1 through collective communication. Ultimately, the first device 400-1 and the second device 400-2 both store the data corresponding to the first device 400-1 and the data corresponding to the second device 400-2, that is, the complete latest data of the first parameter group. The first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups. In response to the completion of data synchronization for the first parameter group, the server 200 is able to obtain the latest data of the complete first parameter group and perform the first forward propagation corresponding to the first model based on the first parameters.

[0037] In some embodiments, in the parameter execution order of the first model, the second parameter group is the parameter group whose execution order in the first model is after the first parameter group. The data synchronization of the second parameter group only needs to rely on the latest data of the second parameter group. The first forward propagation based on the first parameter in the first parameter group does not need to rely on the second parameter group. Therefore, while executing the first forward propagation corresponding to the first model based on the first parameter, the data of the second parameter group can be synchronized. The third device 400-3 stores part of the parameters of the second parameter group of the first model, that is, the third parameter slice, and the fourth device 400-4 stores another part of the parameters of the second parameter group of the first model, that is, the fourth parameter slice. The union of the third parameter slice stored on the third device 400-3 and the fourth parameter slice stored on the fourth device 400-4 is the complete second parameter group. Data synchronization is initiated by the device that stores the parameter slice of the latest data of the second parameter group. For example, when the third parameter slice stored in the third device 400-3 is the latest data of the second parameter group, the data synchronization is initiated by the third device 400-3, that is, the third device 400-3 performs data synchronization on the multiple devices, such as the third device 400-3 and the fourth device 400-4, based on the latest data of the second parameter group. That is, the third device 400-3 sends the stored data to the fourth device 400-4 through collective communication, and the fourth device 400-4 sends the stored data to the third device 400-3 through collective communication. Finally, the third device 400-3 and the fourth device 400-4 both store the data corresponding to the third device 400-3 and the data corresponding to the fourth device 400-4, that is, the complete latest data of the second parameter group.

[0038] It should be noted that the first device 400-1 and the third device 400-3 can be the same device or different devices, and the second device 400-2 and the fourth device 400-4 can be the same device or different devices, and there is no limitation here.

[0039] Furthermore, the first parameter slice stored on first device 400-1 is updated by the central processing unit (CPU). Specifically, when the first parameter slice needs to be updated, first device 400-1 uploads the first parameter slice to the CPU, which then updates it. The CPU then sends the updated first parameter slice to first device 400-1, completing the update of the first parameter slice stored on first device 400-1. The parameter slices stored on second device 400-2, third device 400-3, and fourth device 400-4 are updated in the same manner as first device 400-1 and will not be further described here.

[0040] From the perspective of internal system communication, the embodiments of the present application effectively improve the delay and throughput problems caused by large amounts of data. By storing parameter shards in different devices, the amount of single communication data between devices can be significantly reduced, reducing data transmission delays. Collective communication enables parallel data exchange between devices, improving data throughput per unit time. At the same time, the parallel design of computing and communication enables the system to synchronize subsequent parameters when processing the calculation of preceding parameters, making full use of the communication bandwidth and avoiding the problem of idle communication links in traditional serial modes. The mechanism of the central processing unit centrally updating parameter shards also reduces redundant communications across devices, further optimizes the efficiency of data flow within the system, and alleviates the pressure of large-scale data transmission on throughput performance.

[0041] The distributed data processing method provided in the embodiments of the present application can be applied to image processing scenarios. From the full set of parameters of an image processing model (first model), such as convolution kernel weights and normalization parameters, the first parameter responsible for initial feature extraction and its first parameter group, such as the complete parameter set of the first convolutional layer, are selected. After obtaining the latest data for the first parameter group, synchronization is performed across multiple devices, such as graphics processing unit (GPU) nodes and edge computing devices, via collective communication to ensure consistency in the initial image processing parameters stored on each device. After synchronization of the first parameter group is complete, each device performs a first forward propagation based on the parameters of the first parameter group, such as performing a convolution operation on the image to extract primary features such as edges and textures. Simultaneously, a second parameter group is determined that follows the first parameter group in the execution order of the image processing model (first model), such as the parameters of the batch normalization layer following the convolutional layer and the next convolutional layer. The latest data for the second parameter group is obtained and multi-device synchronization is completed, providing a consistent parameter foundation for further image processing, such as normalization and high-level feature extraction. Through the synchronization and forward propagation of the phased parameter groups, the parallel efficiency of image processing is improved while ensuring computational consistency across multiple devices.

[0042] The distributed data processing method provided in embodiments of the present application can be applied to text processing scenarios. From the full set of parameters of a text processing model (first model), such as word embedding matrices, attention weights, and linear layer weights, the first parameter responsible for encoding initial text features and its corresponding first parameter group are selected. For example, the complete parameter set of the word embedding layer, including the vector matrix corresponding to the vocabulary, is selected. After obtaining the latest data for the first parameter group, synchronization is performed across multiple devices using collective communication to ensure that the initial text processing parameters stored on each device remain consistent. Once synchronization of the first parameter group is complete, each device performs a first forward propagation based on this parameter group. For example, this involves performing a word embedding operation on an input text sequence, such as news sentences or comments, converting each word into a vector of fixed dimension and extracting the basic semantic features of the text, such as the semantic information of the word itself. Simultaneously, a second parameter group is determined for execution after the first parameter group by the text processing model (first model), such as the position encoding layer parameters and attention layer parameters following the word embedding layer. The latest data for the second parameter group is obtained, and multi-device synchronization is completed. This phased parameter group synchronization and forward propagation mechanism can not only ensure the computational consistency of multiple devices during text processing, but also fully leverage the advantages of multi-device parallelism to improve the processing efficiency of tasks such as text classification, sentiment analysis, and machine translation.

[0043] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0044] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the processor provided in an embodiment of the present application, Figure 2 The electronic device 600 shown may be a first device 400-1, a second device 400-2, a third device 400-3, a fourth device 400-4 and a server 200. The electronic device 600 includes: at least one processor 610, a memory 650, at least one network interface 620 and a user interface 630. The various components in the electronic device 600 are coupled together through a bus system 640. It is understood that the bus system 640 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 640 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 640 is not shown in FIG. Figure 2Various buses are labeled as bus system 640 .

[0045] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The user interface 630 includes one or more output devices 631 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 630 also includes one or more input devices 632, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0047] The memory 650 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 650 may optionally include one or more storage devices that are physically remote from the processor 610.

[0048] The memory 650 includes volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 650 described in the embodiments of the present application is intended to include any suitable type of memory.

[0049] In some embodiments, the memory 650 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0050] Operating system 651, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks; A network communication module 652 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 620 . Exemplary network interfaces 620 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB); a presentation module 653 for enabling presentation of information via one or more output devices 631 (e.g., a display screen, a speaker, etc.) associated with the user interface 630 (e.g., a user interface for operating peripheral devices and displaying content and information); The input processing module 654 is configured to detect one or more user inputs or interactions from one of the one or more input devices 632 and to translate the detected inputs or interactions.

[0051] In some embodiments, the processor provided in the embodiments of the present application may be implemented in software. Figure 2 A distributed data processing device 655 stored in memory 650 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a first data synchronization module 6551, a second data synchronization module 6552, a third data synchronization module 6553, a forward propagation execution module 6554, a parameter group determination module 6555, and a parameter allocation module 6556. These modules are logical and can be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0052] In other embodiments, the processor provided in the embodiments of the present application can be implemented in hardware. As an example, the processor provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the distributed data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0053] In some embodiments, the terminal or server can implement the distributed data processing method provided in the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it into a browser environment. In short, the above-mentioned computer-executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module, or plug-in in any form.

[0054] The distributed data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0055] The following describes the distributed data processing method provided by the embodiments of the present application. As previously mentioned, the electronic device that implements the distributed data processing method of the embodiments of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0056] See also Figure 3 , Figure 3 This is a first flow chart of the distributed data processing method provided by the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained, Figure 3 The subject of the step is a device. It should be noted that the distributed data processing method provided in the embodiment of the present application is applied to a distributed system, the distributed system includes multiple devices, the multiple parameters of the first model are divided into multiple parameter groups, and for each parameter group, the parameters of the parameter group are divided into the multiple devices.

[0057] In some embodiments, multiple parameters are divided into multiple parameter groups by the following technical solution: based on the execution order of the multiple parameters, the multiple parameters are sorted to obtain a first parameter sequence; when the parameter amount of the first parameter sequence is not less than the first data amount, data extraction is performed on the first parameter sequence starting from the head, the extracted parameters that meet the first data amount are grouped into a parameter group, and the extracted parameters are removed from the first parameter sequence, wherein the data amount of the parameter group is the first data amount; when the parameter amount of the first parameter sequence is less than the first data amount, the parameters of the first parameter sequence are grouped into a parameter group, wherein the data amount of the parameter group is the data amount of the first parameter sequence.

[0058] As an example, multiple parameters in the model have a clear order when they are executed. This is determined by the dependency between parameters and the model calculation logic. For example, the parameters of the hidden layer in the model are executed first, and the parameters of the output layer in the model are executed later. Therefore, when extracting parameters to obtain parameter groups, it is necessary to ensure that the execution order between parameters remains unchanged. Therefore, before extracting parameters, multiple parameters need to be sorted in the execution order. The position of the parameter in the first parameter sequence obtained represents the order in which the parameter is executed. The earlier the parameter is executed, the closer it is to the head of the first parameter sequence. For example, parameter The position in the first parameter sequence is , then the parameter It is The first data size is the maximum amount of data that each parameter group can accommodate. When the number of parameters in the first parameter sequence is no less than the first data size, the first parameter sequence can be extracted into at least two parameter groups. Data extraction must then begin at the head of the first parameter, grouping the extracted parameters no larger than the first data size into a single parameter group. The extracted parameters are then removed from the first parameter sequence, allowing data extraction to resume at the head of the first parameter until no parameters remain in the first parameter sequence. This means that all parameters in the first parameter sequence have corresponding parameter groups. When the number of parameters in the first parameter sequence is less than the first data size, the first parameter sequence (with a parameter size less than the first data size) corresponds to only one parameter group. Therefore, the parameters in the first parameter sequence must be grouped into a single parameter group. The number of parameter groups in this case equals the data size of the first parameter sequence. For example, in an image processing model (first model), the order of parameter execution is determined by the dependencies between network layers and the image computation logic. For example, convolutional layer parameters are first used for feature extraction, followed by pooling and fully connected layer parameters. Therefore, this order must be maintained when extracting parameters into parameter groups. Before extraction, they must be sorted in execution order to obtain the first parameter sequence. The position of the parameters in the first parameter sequence represents the execution order, and the closer to the front, the earlier they participate in image processing. The first data amount is the maximum capacity of each parameter group. If the parameter amount of the first parameter sequence is not less than the first data amount, parameters that are not greater than the first data amount are extracted from the head to form a parameter group, and the extracted parameters are removed from the first parameter sequence. The remaining parameters are extracted in this way until there are no parameters in the first parameter sequence; if the parameter amount of the first parameter sequence is less than the first data amount, the entire first parameter sequence is the parameter group. The parameters in the first parameter sequence are the parameters after flattening. The flattening process requires obtaining the data in the parameters according to the first rule, and splicing the data in the obtained parameters to obtain the flattened parameters. For example, for the parameter For example, the first rule is to read line by line, then the data obtained by reading line by line is [1,2], [3,4], and [1,2], [3,4] are spliced ​​into , which is the parameter Perform flattening to obtain the parameters after flattening .

[0059] In some embodiments, grouping the parameters of the first parameter sequence into a parameter group can be achieved through the following technical solution: grouping the parameters of the first parameter sequence into a first initial parameter group; in response to the data amount of the first initial parameter group not being divisible by a first threshold value, adding a virtual parameter to the first initial parameter group so that the first initial parameter group after adding the virtual parameter can be divisible by the first threshold value, and using the first initial parameter group to which the virtual parameter is added as the parameter group.

[0060] As an example, since the parameters in each parameter group need to be evenly distributed to multiple devices to ensure that the time for each device to complete the task is basically the same, it is avoided that the overall progress is slowed down due to the heavy load of some devices, or some devices are idle and cause waste of resources. Here, the number of devices is the first threshold, so it is necessary to ensure that the amount of data in the parameter group can be divided by the first threshold. The specific steps are: group the parameters of the first parameter sequence into a first initial parameter group; in response to the fact that the amount of data in the first initial parameter group cannot be divided by the first threshold, it is necessary to add virtual parameters to the first initial parameter group so that the first initial parameter group with the added virtual parameters can be divided by the first threshold, and the first initial parameter group with the added virtual parameters is used as the parameter group. For example, if the first threshold is 3, the first initial parameter group is , the data volume of the first initial parameter group is 5, at this time 5 3=1 remainder 2, the data volume of the first initial parameter group cannot be divided by the first threshold, so it is necessary to add a virtual parameter to the first initial parameter group, such as , so that the data volume of the first initial parameter group after adding the dummy parameter can be divided by the first threshold. Here, the minimum data volume that can be divided by the first threshold is 6. Therefore, only one dummy parameter needs to be added to the first initial parameter group. That is, the first initial parameter group after adding the dummy parameter is , the data volume of the first initial parameter group after adding the virtual parameter is 6, which can be divided by the first threshold, that is, the first initial parameter group with the virtual parameter added is is a parameter group. In the image processing scenario, the first parameter sequence sorted in execution order is first divided and processed to obtain a first initial parameter group. The parameters in the first initial parameter group may include parameters related to image feature extraction, such as convolution kernel weights and bias terms. When the data volume of the first initial parameter group cannot be divided by the first threshold, for example, the first threshold is the basic unit size of parallel computing of the device in image processing, it is necessary to add virtual parameters to the first initial parameter group, such as These virtual parameters do not participate in the actual image feature calculation and are only used to fill in the data structure so that the first initial parameter group after filling in, that is, the parameter group, can be divided by the first threshold value. Finally, a parameter group that can be directly used for parallel processing of the image model is formed, ensuring that multiple devices can achieve uniform distribution of data blocks and efficient parallel computing when processing image data.

[0061] In some embodiments, grouping the extracted parameters into a parameter group can be achieved through the following technical solution: grouping the extracted parameters into a second initial parameter group; in response to the data volume of the second initial parameter group not being divisible by the first threshold, adding a virtual parameter to the second initial parameter group so that the second initial parameter group with the added virtual parameter can be divided by the first threshold, and the second initial parameter group with the added virtual parameter is used as the parameter group.

[0062] As an example, since the parameters in each parameter group need to be evenly distributed to multiple devices to ensure that the time for each device to complete the task is basically the same, it is avoided that the overall progress is slowed down due to the heavy load of some devices, or some devices are idle and cause waste of resources. Here, the number of devices is the first threshold, so it is necessary to ensure that the amount of data in the parameter group can be divided by the first threshold. The specific steps are: the extracted parameters are formed into a second initial parameter group; in response to the data amount of the second initial parameter group not being divisible by the first threshold, it is necessary to add virtual parameters to the second initial parameter group so that the second initial parameter group with the added virtual parameters can be divided by the first threshold, and the second initial parameter group with the added virtual parameters is used as the parameter group. For example, if the first threshold is 3 and the second initial parameter group is , the data volume of the second initial parameter group is 5, at this time 5 3=1 remainder 2, the data volume of the second initial parameter group cannot be divided by the first threshold, so it is necessary to add a virtual parameter to the second initial parameter group, such as , so that the data volume of the second initial parameter group after adding the dummy parameter can be divided by the first threshold. Here, the minimum data volume that can be divided by the first threshold is 6. Therefore, only one dummy parameter needs to be added to the second initial parameter group. That is, the second initial parameter group after adding the dummy parameter is , the data volume of the second initial parameter group after adding the virtual parameter is 6, which can be divided by the first threshold, that is, the second initial parameter group with the virtual parameter added is is a parameter group. In the image processing scenario, the first parameter sequence sorted in execution order is first divided and processed to obtain a second initial parameter group. The parameters in the second initial parameter group may include parameters related to image feature extraction, such as convolution kernel weights and bias terms. When the amount of data in the second initial parameter group cannot be divided by the first threshold, for example, the first threshold is the basic unit size of device parallel computing in image processing, it is necessary to add virtual parameters to the second initial parameter group, such as These virtual parameters do not participate in the actual image feature calculation and are only used to fill in the data structure so that the second initial parameter group after filling in, that is, the parameter group, can be divided by the first threshold. Ultimately, a parameter group that can be directly used for parallel processing of the image model is formed, ensuring that multiple devices can achieve uniform distribution of data blocks and efficient parallel computing when processing image data.

[0063] Through the embodiments of the present application, dummy parameters can be added to the initial parameter group, and the resulting parameter group can be divided by the first threshold. Since the embodiments of the present application need to evenly distribute the parameters in each parameter group to multiple devices, where the first threshold is the number of devices, adding dummy parameters to the initial parameter group can ensure that the resulting parameter group is divisible by the first threshold, so as to achieve the purpose of even distribution to multiple devices, thereby achieving the basic consistency of the amount of parameter data stored on each device, so that the time it takes for the devices to complete the task is also basically the same, avoiding slowing down the overall progress due to overloading of some devices, or wasting resources due to idleness of some devices.

[0064] Through the embodiments of the present application, the parameter execution order is strictly followed to ensure the consistency and correctness of the model logic, and the strategy for obtaining the corresponding parameter group can be determined based on the size relationship between the parameter quantity of the first parameter sequence and the first data quantity of the parameter group, so as to adapt to the first parameter sequence with different parameter quantities, thereby further improving the accuracy and rationality of determining the parameter group based on the first parameter sequence.

[0065] In some embodiments, each device is allocated some parameters of the first model. Allocation of some parameters of the first model to each device can be achieved by the following technical solution: evenly distributing data of all parameters included in the parameter group to the multiple devices.

[0066] Through the embodiments of the present application, the data of all parameters included in each parameter group are evenly distributed to multiple devices, which can ensure that the amount of parameter data stored on each device is basically consistent, so that the time for the devices to complete tasks is also basically consistent, avoiding the situation where the overall progress is slowed down due to excessive load on some devices when the devices are executing tasks, or some devices are idle, causing waste of resources.

[0067] In step 101 , data synchronization is performed on the plurality of devices based on the latest data of a first parameter group to which a first parameter to be executed belongs.

[0068] As an example, the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups.

[0069] In some embodiments, Figure 3Step 101 shown can be implemented by the following technical solution: the parameters in the first parameter group are assigned to the multiple devices. Each of the devices performs the following processing: synchronizing the latest data of the parameters in the first parameter group assigned to the device to other devices, where the other devices are devices other than the device in the multiple devices.

[0070] As an example, the parameters in the first parameter group are distributed to multiple devices, specifically: the first device stores part of the parameters of the first parameter group of the first model, that is, the first parameter slice, and the second device stores another part of the parameters of the first parameter group of the first model, that is, the second parameter slice. The union of the first parameter slice stored on the first device and the second parameter slice stored on the second device is the complete first parameter group. Data synchronization is initiated by the device that stores the parameter slice of the latest data of the first parameter group. For example, when the first parameter slice stored on the first device is the latest data of the first parameter group, the first device initiates data synchronization. That is, the first device performs data synchronization on multiple devices, such as the first device and the second device, based on the latest data of the first parameter group to which the first parameter to be executed belongs. That is, the first device sends the stored parameters to the second device via collective communication, and the second device sends the stored parameters to the first device via collective communication. Ultimately, the first device and the second device both store the parameters corresponding to the first device and the parameters corresponding to the second device, that is, the latest data of the complete first parameter group. The first parameter slice stored on the first device is updated by the central processing unit CPU, that is, when the first parameter slice needs to be updated, the first device will upload the first parameter slice that needs to be updated to the central processing unit, which will update it. The central processing unit will send the updated first parameter slice to the first device, and the update of the first parameter slice stored on the first device is completed. The update of the parameter slice stored on the second device is consistent with that of the first device, which will not be repeated here. In the image processing scenario, when the first device holds the latest data of the first parameter slice assigned to the first device in the first parameter group, such as a convolution layer parameter, the first device needs to synchronize the latest data of the first parameter slice stored in itself to other devices participating in image processing, such as the second device. For example, after the first device responsible for extracting image edge features stores the latest convolution kernel parameters, the first device will send these latest convolution kernel parameters to the second device that also participates in the feature calculation of this layer to ensure that all devices use consistent parameters for image convolution, pooling and other operations, avoid feature extraction deviations caused by inconsistent parameters, and ensure the calculation consistency and result accuracy when multiple devices collaborate to process images.

[0071] Through the embodiments of the present application, synchronization of parameters between various devices allows multiple devices to quickly obtain the latest data of the complete first parameter group, avoiding deviations in forward propagation due to only partial parameters remaining on the device.

[0072] In some embodiments, Figure 3 The shown step 101 can also be implemented by the following technical solution: in response to the latest data of the first parameter group to which the first parameter to be executed belongs not being synchronized to the multiple devices, data synchronization is performed on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs.

[0073] As an example, the condition for triggering data synchronization is that the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to multiple devices, that is, the data stored on multiple devices are inconsistent in version or are not the latest data of the complete first parameter group. In this case, data synchronization needs to be performed on multiple devices, specifically: the first device stores part of the parameters of the first parameter group of the first model, that is, the first parameter shard, and the second device stores another part of the parameters of the first parameter group of the first model, that is, the second parameter shard. The union of the first parameter shard stored on the first device and the second parameter shard stored on the second device is the complete first parameter group. Data synchronization is initiated by the device that stores the parameter slice of the latest data of the first parameter group. For example, when the first parameter slice stored by the first device is the latest data of the first parameter group, the data synchronization is initiated by the first device, that is, the first device performs data synchronization on multiple devices, such as the first device and the second device, based on the latest data of the first parameter group to which the first parameter to be executed belongs. That is, the first device sends the stored parameters to the second device through collective communication, and the second device sends the stored parameters to the first device through collective communication. Finally, the first device and the second device both store the parameters corresponding to the first device and the parameters corresponding to the second device, that is, the latest data of the complete first parameter group. The update of the parameter slices stored on the third and fourth devices is consistent with that of the first device, and will not be repeated here. In the image processing scenario, when the first device holds the latest data of the first parameter slice assigned to the first device in the first parameter group, such as a convolutional layer parameter, the first device needs to synchronize the latest data of the first parameter slice stored by itself to other devices participating in image processing, such as the second device. For example, after the first device responsible for extracting image edge features stores the latest convolution kernel parameters, the first device will send these latest convolution kernel parameters to the second device that also participates in the feature calculation of this layer to ensure that all devices use consistent parameters for image convolution, pooling and other operations, avoid feature extraction deviations caused by inconsistent parameters, and ensure calculation consistency and result accuracy when multiple devices collaborate to process images.

[0074] Through the embodiment of the present application, data synchronization can be triggered when the latest data of the first parameter group to which the first parameter to be executed belongs has not been synchronized to multiple devices, which can ensure that all devices hold the latest data of the first parameter group that is completely consistent and complete before executing the task. This on-demand triggering of synchronization processing can avoid repeated synchronization operations when the latest data has been synchronized, reduce unnecessary bandwidth usage, device computing power consumption and time loss, and allow resources to be concentrated on scenarios that really need synchronization, thereby improving overall processing efficiency; and only triggering operations when the latest data is not synchronized can accurately ensure that all devices hold the latest data of the first parameter group that is completely consistent and complete before executing the task, eliminate execution deviations caused by information missing, and ensure the accuracy of the results and process stability of multi-device collaborative tasks.

[0075] Continue to see Figure 3 In step 102, in response to completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameters.

[0076] As an example, after the data synchronization of the first parameter group is completed, all devices have the latest data of the complete first parameter group, so the first parameter in the first parameter group is also the latest first parameter. At this time, it is necessary to perform the first forward propagation of the corresponding first model based on the first parameter. In the image processing scenario, when the first parameter group, such as the parameters of a convolution layer or pooling layer, is synchronized between multiple devices, each device already has the complete and consistent latest parameters of the first parameter group. At this time, the system immediately executes the first forward propagation of the first model based on the first parameter group, such as feature extraction of the input image: through the synchronized first parameter group, such as the convolution kernel parameters and the image pixel matrix, convolution operations are performed to generate basic features such as edges and textures, or the feature map dimensions are compressed through pooling operations. This process ensures that multiple devices can perform calculations based on completely consistent parameters, ensuring the consistency of feature extraction results.

[0077] In some embodiments, Figure 3 After step 102, the following technical solution may be performed: the first parameter group further includes a second parameter. A third forward propagation corresponding to the first model is performed based on the second parameter and the first forward propagation result corresponding to the first forward propagation to obtain a third forward propagation result.

[0078] As an example, the first parameter group is derived from multiple parameter groups, and the multiple parameter groups are obtained based on a first parameter sequence determined based on an execution order. Therefore, the parameters in the first parameter group have a clear execution order, and the execution order of the second parameter in the first parameter group is after the execution order of the first parameter in the first parameter group. Therefore, after executing the first forward propagation corresponding to the first model based on the first parameters and obtaining the first forward propagation result, it is necessary to execute the third forward propagation corresponding to the first model based on the first forward propagation result and the second parameters to obtain the third forward propagation result. In the image processing scenario, in addition to the parameters involved in the initial feature extraction, the first parameter group also includes second parameters, such as advanced convolution layer weights, activation function parameters, etc. After completing the first forward propagation based on the first parameters, such as extracting basic features such as image edges and textures, the system will use the second parameters and the basic feature map obtained by the first forward propagation to execute the third forward propagation. For example, the advanced convolution kernel corresponding to the second parameter is used to perform deeper feature fusion and abstraction on the basic feature map to generate an advanced feature map containing information such as target contour and local semantics, providing more discriminative feature support for subsequent image classification, target detection and other tasks. The final third forward propagation result can be directly used for image recognition decision-making or further feature processing.

[0079] In this embodiment, the first forward propagation is performed using the first parameter, and then the third forward propagation is performed by combining the first forward propagation result with the second parameter. This design reduces the parameter calculation pressure of a single forward propagation, allowing the second parameter to specifically supplement the feature gaps of the first forward propagation, reducing information redundancy and loss, and more accurately extracting deep data correlations, thereby improving the effectiveness of the third forward propagation result and enhancing the accuracy and stability of the model processing task.

[0080] In step 103 , data synchronization is performed on the plurality of devices based on the latest data of the second parameter group.

[0081] As an example, the second parameter group is a parameter group that is executed after the first parameter group in the first model. It should be noted that step 102 and step 103 are executed simultaneously. The data synchronization of the second parameter group only needs to rely on the latest data of the second parameter group. The first forward propagation based on the first parameter in the first parameter group does not need to rely on the second parameter group. Therefore, while the first forward propagation corresponding to the first model is executed based on the first parameter, the data of the second parameter group can be synchronized. The specific steps of performing data synchronization on multiple devices based on the latest data of the second parameter group are as follows: the third device stores part of the parameters of the second parameter group of the first model, that is, the third parameter slice, and the fourth device stores another part of the parameters of the second parameter group of the first model, that is, the fourth parameter slice. The union of the third parameter slice stored on the third device and the fourth parameter slice stored on the fourth device is the complete second parameter group. When the latest data of the second parameter group has not been synchronized to the multiple devices, data synchronization is triggered. Data synchronization is initiated by the device that stores the parameter slice of the latest data of the second parameter group. For example, when the third parameter slice stored by the third device is the latest data of the second parameter group, the data synchronization is initiated by the third device, that is, the third device performs data synchronization on multiple devices, such as the third device and the fourth device, based on the latest data of the second parameter group. That is, the third device sends the stored data to the fourth device through collective communication, and the fourth device sends the stored data to the third device through collective communication. Ultimately, the third device and the fourth device both store the data corresponding to the third device and the data corresponding to the fourth device, that is, the complete latest data of the second parameter group. The third device and the first device in step 101 can be the same device or different devices, and the fourth device and the second device in step 101 can be the same device or different devices. There is no limitation here. In the image processing scenario, when the third device holds the latest data of the first parameter slice assigned to the third device in the second parameter group, such as the parameters of a convolutional layer, the third device needs to synchronize the latest data of the third parameter slice stored by itself to other devices participating in image processing, such as the fourth device. For example, after the third device responsible for extracting image edge features stores the latest convolution kernel parameters, the third device will send these latest convolution kernel parameters to the fourth device that also participates in the feature calculation of this layer to ensure that all devices use consistent parameters for image convolution, pooling and other operations, avoid feature extraction deviations caused by inconsistent parameters, and ensure calculation consistency and result accuracy when multiple devices collaborate to process images.

[0082] From the perspective of internal system communication, the embodiments of the present application effectively improve the delay and throughput problems caused by large amounts of data. By storing parameter shards in different devices, the amount of single communication data between devices can be significantly reduced, reducing data transmission delays. The collective communication method realizes parallel data exchange between devices, improving the data throughput per unit time. At the same time, the parallel design of computing and communication enables the system to synchronize subsequent parameters when processing the calculation of pre-order parameters, making full use of the communication bandwidth and avoiding the problem of idle communication links in traditional serial modes. The mechanism of the central processing unit to centrally update parameter shards also reduces redundant communications across devices, further optimizes the efficiency of data flow within the system, and alleviates the pressure of large-scale data transmission on throughput performance.

[0083] In some embodiments, Figure 3 After step 103 shown, see Figure 4 , you can also perform steps 104 to 105, the following combination Figure 4 Provide explanation.

[0084] In step 104 , in response to the completion of data synchronization for the second parameter group, data synchronization is continued on the multiple devices based on the latest data of the third parameter group until all parameter groups are synchronized on the multiple devices.

[0085] In step 105 , in response to the completion of the first forward propagation, a second forward propagation corresponding to the first model is performed based on the second parameter group until the forward propagation is completed based on all parameter groups.

[0086] As an example, the third parameter group is a parameter group whose execution order is after the second parameter group. In this embodiment of the present application, data synchronization is performed on multiple devices for the latest data of each parameter group in sequence according to the execution order of the parameter groups until all parameter groups are synchronized on multiple devices. Furthermore, after the first forward propagation based on the parameters in the first parameter group is completed, forward propagation corresponding to the first model is performed on each parameter group in sequence according to the execution order of the parameter groups until all parameter groups are synchronized. Taking the third parameter group and the second parameter group as an example, the third parameter group is the parameter group whose execution order is after the second parameter group. Since, during the data synchronization process for the third parameter group, the data synchronization of the third parameter group only relies on the latest data of the third parameter group and does not need to rely on the second parameter group, the second forward propagation for the first model based on the second parameter group does not need to rely on the third parameter group. Therefore, the data of the third parameter group can be synchronized while the second forward propagation for the first model based on the second parameter group is performed. In other words, in this embodiment of the present application, forward propagation and data synchronization can be performed simultaneously, and the execution order of the parameter group targeted by the forward propagation is before the execution order of the parameter group targeted by the data synchronization. In image processing scenarios, after the second parameter group, such as pooling layer parameters, is synchronized across multiple devices, the third parameter group, such as cross-device data synchronization of fully connected layer weights, is immediately started, and this process is continued until all parameter groups, including parameters of each convolutional layer, normalization layer, etc., are synchronized across multiple devices. After the first forward propagation, such as bottom-level edge feature extraction, is completed, the synchronized second parameter group is immediately used to perform the second forward propagation, such as mid-level texture feature fusion, and the execution order of the parameter groups is followed until all parameter groups have participated in the forward propagation. This ultimately completes the extraction of high-level semantic features from the original image, such as target categories and location information, ensuring efficient connection between calculation and synchronization in the image processing process and improving the overall efficiency of model inference or training.

[0087] Through the embodiments of the present application, multi-device parameter group synchronization is carried out in batches according to the execution order. After the data synchronization of the previous parameter group, such as the second parameter group, is completed, the data synchronization of the subsequent parameter group, such as the third parameter group, is carried out. This ensures the consistency of multi-device parameters. At the same time, during the forward propagation of the model, the execution is carried out step by step according to the execution order of the parameter groups, ensuring the orderly connection of the forward propagation. This not only avoids conflicts in parameter group data synchronization, but also allows the model operation to proceed in order, reducing redundant interactions.

[0088] In some embodiments, see Figure 5 , based on Figure 4 After the second forward propagation result of the second forward propagation is obtained in step 105, steps 106 to 111 may be performed. Figure 5 Provide explanation.

[0089] In step 106, the training loss of the first model is determined based on the second forward propagation result corresponding to the second forward propagation.

[0090] As an example, the second forward propagation result is a prediction result, and the training loss of the first model can be determined by determining the cross entropy loss or mean square loss between the prediction result, that is, the second forward propagation result and the true result.

[0091] In step 107, a first gradient corresponding to the second parameter group is determined based on the training loss.

[0092] In step 108 , the second parameter set is updated based on the first gradient.

[0093] As an example, the backpropagation algorithm uses the chain rule to calculate the partial derivative of the second parameter set with respect to the training loss, i.e., the first gradient. This first gradient reflects the degree to which changes in the second parameter set affect the training loss. Then, combined with the optimizer, the second parameter set is adjusted based on the magnitude and direction of the training loss, reducing the training loss. This process is repeated until the training loss converges to a relatively optimal value. At this point, the corresponding second parameter set is considered the optimal second parameter set, and the second parameter set is updated to the optimal second parameter set.

[0094] In step 109 , based on the first gradient corresponding to the second parameter group, a second gradient corresponding to the first parameter group is determined, so as to update the first parameter group based on the second gradient.

[0095] As an example, since the execution order of the second parameter group is after the first parameter group during forward propagation, the second gradient of the first parameter group needs to depend on the first gradient of the second parameter group. Therefore, it is necessary to determine the second gradient corresponding to the first parameter group based on the first gradient corresponding to the second parameter group, specifically: multiply the first gradient by the derivative of the connection relationship between the first parameter group and the second parameter group, and calculate the partial derivative of the first parameter group on the training loss, that is, the second gradient. The second gradient can reflect the degree of influence of the change of the first parameter group on the training loss. Then, in combination with the optimizer, the first parameter group is adjusted according to the size and direction of the training loss, so that the training loss changes in the direction of reduction. Repeat this process until the training loss converges to a better value. At this time, the corresponding first parameter group is the optimal first parameter group, and the first parameter group is updated to the optimal first parameter group.

[0096] In step 110 , in response to the completion of updating the second parameter group, data synchronization is performed on the plurality of devices based on the updated second parameter group.

[0097] In step 111 , in response to the completion of updating the first parameter group, data synchronization is performed on the plurality of devices based on the updated first parameter group.

[0098] As an example, after the second parameter group is updated, data synchronization needs to be performed on multiple devices so that the updated second parameter group is stored on all devices. After the first parameter group is updated, data synchronization needs to be performed on multiple devices so that the updated first parameter group is stored on all devices. The steps for performing data synchronization can be found in Figure 3 Step 101 is shown and will not be described here in detail. It should be noted that, since the execution order of the first parameter group is before the second parameter group during forward propagation, the above step 111 needs to be executed before the above step 110, that is, it is necessary to first execute data synchronization of the updated first parameter group on multiple devices, and then the first model executes forward propagation based on the updated first parameter group while executing data synchronization of the updated second parameter group on multiple devices. The reason why forward propagation and data synchronization are executed simultaneously is that the data synchronization of the second parameter group is only related to the updated second parameter group, and the forward propagation of the first parameter group is only related to the first parameter group after data synchronization, so forward propagation and data synchronization can be performed simultaneously.

[0099] For example, in an image processing scenario, the result of the second forward propagation could be a predicted image classification label, such as "cat" or "dog." By calculating the cross-entropy loss between the predicted result and the true image label, the model training loss is determined to measure prediction bias. Based on the training loss, the second parameter group is reversely derived, such as the first gradient of the fully connected layer parameters, reflecting the degree to which these layer parameters affect the loss. Using the first gradient, the second parameter group is updated using algorithms such as gradient descent to reduce classification error. The first gradient of the second parameter group is used to derive the first parameter group, such as the second gradient of the underlying convolutional layer. This is then used to update the first parameter group, such as the convolution kernel, to ensure that the underlying feature extraction more closely matches the true image features. After the second and first parameter groups are updated, the updated parameters are immediately synchronized across multiple devices, ensuring that all devices use the same latest parameters in subsequent training, maintaining the stability and convergence efficiency of image processing model training.

[0100] Through the embodiment of the present application, the model training loss is first accurately calculated with the second forward propagation result, and then the first gradient of the second parameter group is determined based on the training loss and the update is completed. At the same time, the second gradient of the first parameter group is derived based on the first gradient to realize the update of the first parameter group, which can ensure that the update of the parameter group follows the sequence dependency of the forward propagation. After the first parameter group and the second parameter group are updated respectively, data synchronization can be performed in a timely manner, and the updated first parameter group and the second parameter group can be synchronized to multiple devices, which not only avoids the unupdated parameter group from interfering with the collaborative operation of multiple devices, but also ensures that the parameter groups of each device are always consistent and the latest data, effectively improving the stability of model training in a multi-device environment, the efficiency of parameter group synchronization and the final training effect.

[0101] Through the embodiment of the present application, data synchronization is performed on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, wherein the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups; by synchronizing the data of the first parameter group on multiple devices in a targeted manner, on-demand synchronization can be achieved, the amount of data transmission can be reduced, and the synchronization efficiency can be improved. In response to the completion of the data synchronization for the first parameter group, the first forward propagation corresponding to the first model is performed based on the first parameter, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is a parameter group whose execution order in the first model is after the first parameter group. After the data synchronization of the first parameter group is completed, the forward propagation based on the first parameter in the first parameter group is immediately started, and the data of the second parameter group is synchronized on multiple devices at the same time, so that the forward propagation calculation and the parameter group synchronization are carried out in parallel, which reduces the waiting time and improves the overall efficiency. In the parameter execution sequence of the first model, the second parameter group is ranked after the first parameter group. Synchronization of the second parameter group only needs to rely on the latest data of the second parameter group, and forward propagation based on the first parameter in the first parameter group does not need to rely on the second parameter group. Therefore, the parallel processing between data synchronization of the second parameter group and forward propagation based on the first parameter in the first parameter group does not affect the accuracy of the current calculation. This embodiment of the present application achieves the parallel execution of forward propagation and parameter group data synchronization, improving data processing efficiency.

[0102] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0103] In traditional data-parallel training, all processors must store the complete model (first model) state, including model parameters, gradients, and optimizer state. This redundant storage causes graphics memory usage to grow with the number of processors, severely limiting the trainable scale of large models. Sharding can be used to distribute the model state, significantly reducing graphics memory usage. Its three-level optimization strategy is as follows: First level: only the optimizer state is sharded, and each device stores the complete parameters and gradients; Second level: Sharded optimizer state and gradients, with each device storing complete parameters; Level 3: Shard optimizer status, gradients, and parameters, minimizing memory usage.

[0104] However, during distributed optimizer training, after parameter updates, all-gather operations are required to synchronize the shard parameters of each graphics processing unit (GPU) within the data parallel (DP) domain. Because communication within the data parallel domain typically requires communication across compute nodes, meaning that different machines must communicate, this process introduces significant communication latency, becoming a bottleneck for training efficiency.

[0105] In the mainstream technical solutions in the industry, some training frameworks can also support full network coverage, but they can only support the first level mentioned above. Although some frameworks can support segmentation from the first to the third level, they cannot achieve full network coverage. Therefore, the method provided in the embodiment of the present application is to solve the performance bottleneck problem caused by the use of full network communication when updating parameters of the distributed optimizer. The embodiment of the present application can support communication coverage when updating parameters of the distributed optimizer, and can update parameters asynchronously, without having to wait for all gradients to be obtained before updating parameters.

[0106] The optimizer usually contains multiple parameters, these parameters and their corresponding gradients and optimizer states. Each parameter is flattened and passed to the optimizer. Each parameter here needs to correspond to multiple values. For example, the number of values ​​in parameter 1 is 1024. The flattening process needs to obtain the data in the parameters according to the first rule and splice the obtained parameter data to obtain the flattened parameters (first parameter sequence). For example, for parameter For example, the first rule is to read line by line, then the data obtained by reading line by line is [1,2], [3,4], and [1,2], [3,4] are spliced ​​into , which is the parameter Perform flattening to obtain the parameters after flattening See also Figure 6 , Figure 6 It is a schematic diagram of the parameters after flattening provided in an embodiment of the present application.

[0107] The flattened parameters need to be bucketed (parameter groups) and the bucket size set. When the number of parameters in a bucket reaches the upper limit of the bucket (the first data size), a new bucket is created and the remaining parameters are placed in the new bucket. If the sum of the number of parameters in a bucket cannot be divided by the size of the data parallel domain (the first threshold), the data in the bucket is padded, for example, with a value of 0 (dummy parameter). After padded, the sum of the number of parameters in the bucket is an integer multiple of the size of the data parallel domain. See Figure 7 , Figure 7 This is a schematic diagram of parameter bucketing provided in an embodiment of the present application.

[0108] See also Figure 8 , Figure 8 This is a schematic diagram of device allocation provided by an embodiment of the present application. The parameters in each bucket are evenly distributed to multiple devices corresponding to the data parallel domain. The parameters are downloaded to the central processing unit (CPU) for update. The updated parameters are sent by the CPU to the devices corresponding to the parameters. The updated optimizer state is also saved by each device. Each bucket is updated as a parameter group updated by the optimizer. Since the parameter update is an element-by-element operation, the partitioned update of the parameters has no effect on the accuracy. See Figure 8 , each bucket is evenly divided among the devices in the data parallel domain. For example, the first part of parameter 1 in bucket 1 is allocated to device 0, the last part of parameter 1, all of parameter 2, and the first part of parameter 3 are allocated to device 1, and the last part of parameter 3 is allocated to device 2. In this way, each bucket is traversed. When the specified bucket is traversed, the parameters are downloaded to the central processor. The central processor passes the updated parameters to each device corresponding to the parameters and updates the parameters of the corresponding parts. Each device is equal, and the devices communicate with each other through collective communication. Forward propagation, backward propagation, and parameter update are performed on each device. The data on each device is synchronized during the next forward propagation after the parameter update. See Figure 9 , Figure 9 This is the first schematic diagram of parameter update provided by the embodiment of the present application. In the related art, all parameters are usually collected and updated synchronously at one time before forward propagation is performed. This approach will cause long-term communication blockage.

[0109] See also Figure 10 , Figure 10This is the second schematic diagram of parameter updates provided by an embodiment of the present application. After forward propagation begins, when executing each trainable parameter, the embodiment of the present application determines whether the data on all devices in the bucket to which the parameter belongs is synchronized (data synchronization). If not, a full collection is called to synchronize the data on all devices in the bucket to which the parameter belongs. After synchronization is complete, forward propagation is performed again, and a communication stream is created. Within this communication stream, the next bucket is synchronized. At this point, the communication stream and the propagation stream are synchronized, completing the masking.

[0110] See also Figure 10 When executing parameter 1 (the first parameter), since the data of each device in bucket 1 (the first parameter group) has not been synchronized at this time, a full collection is called to synchronize each device in bucket 1. After synchronization is complete, parameter 1 is executed again. At the same time, a communication flow is created in which the devices in bucket 2 (the second parameter group) are synchronized in advance. When executing parameters 2 / 3, since parameters 2 / 3 have already been synchronized in bucket 1, forward propagation is performed directly. When executing parameter 4, since bucket 2 has already been synchronized, forward propagation is performed directly without waiting. And so on, completing the model forward propagation and parameter synchronization.

[0111] Since parameter synchronization and update are grouped into buckets, there is no need to wait for all backpropagation to be completed before updating and synchronizing the parameters. When the parameters in a bucket have completed backpropagation, they can be downloaded to the central processing unit for early update.

[0112] See also Figure 11 , Figure 11 This is the third schematic diagram of parameter update provided by the embodiment of the present application. In the conventional distributed training process, the conventional process is to perform forward propagation, backward propagation, parameter update on a single device, and synchronize the updated parameters to other devices. A bucket contains multiple parameters. During the training process, the forward propagation of bucket 1 / 2 is executed in sequence, and after the loss is obtained, the backward propagation of bucket 2 / 1 is started. After the gradients of all parameters are obtained, the optimizer is called for update. After the parameters of a device are updated, the parameters on all devices are synchronized using full collection, that is, each device needs to know the data on other devices. When the synchronization is completed, the next round of forward propagation begins.

[0113] See also Figure 12 , Figure 12This is the fourth schematic diagram of parameter update provided by the embodiment of the present application. When the gradient calculation in bucket 2 is completed during back propagation, the parameters are downloaded to the central processor for update, and the back propagation of bucket 1 and parameter update are continued. When each device in buckets 1 and 2 completes their respective parameter updates, forward propagation begins. When bucket 1 performs the next round of forward propagation, it is necessary to first perform a full collection for parameter synchronization, and then start the forward propagation of bucket 1. At the same time, a full collection is performed on the communication stream to synchronize the parameters of bucket 2. See Figure 13 , Figure 13 This is a timeline diagram for parameter updates provided by an embodiment of the present application. During backward propagation, the computations performed by bucket 1 on the device and the parameter updates performed by bucket 2 on the CPU can be masked. During the next forward propagation, the parameter updates performed by bucket 2 on the communication flow can be masked by the forward propagation performed by bucket 1 on the device.

[0114] It is understandable that in the embodiments of the present application, when parameters and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0115] The following continues to describe the exemplary structure of the distributed data processing device 655 provided in the embodiment of the present application implemented as a software module. In some embodiments, see Figure 2 , the software modules stored in the distributed data processing device 655 of the memory 650 may include: A first data synchronization module 6551 is configured to synchronize data on the multiple devices based on the latest data of a first parameter group to which a first parameter to be executed belongs, wherein the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups; The second data synchronization module 6552 is used to perform a first forward propagation corresponding to the first model based on the first parameters in response to the completion of data synchronization for the first parameter group, and to perform data synchronization on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is a parameter group whose execution order in the first model is after the first parameter group.

[0116] In some embodiments, the device also includes a third data synchronization module 6553, which is used to, in response to the completion of data synchronization for the second parameter group, continue to perform data synchronization on the multiple devices based on the latest data of the third parameter group until all parameter groups are synchronized on the multiple devices, wherein the third parameter group is the parameter group whose execution order is after the second parameter group; in response to the completion of the first forward propagation, perform the second forward propagation corresponding to the first model based on the second parameter group until the forward propagation is completed based on all parameter groups.

[0117] In some embodiments, the third data synchronization module 6553 is also used to determine the training loss of the first model based on the second forward propagation result corresponding to the second forward propagation; determine the first gradient corresponding to the second parameter group based on the training loss; update the second parameter group based on the first gradient, and determine the second gradient corresponding to the first parameter group based on the first gradient corresponding to the second parameter group, so as to update the first parameter group based on the second gradient; in response to the completion of the update of the second parameter group, perform data synchronization on the multiple devices based on the updated second parameter group; in response to the completion of the update of the first parameter group, perform data synchronization on the multiple devices based on the updated first parameter group.

[0118] In some embodiments, the first data synchronization module 6551 is also used to assign the parameters in the first parameter group to the multiple devices, and perform the following processing through each of the devices: synchronize the latest data of the parameters in the first parameter group assigned to the device to other devices, wherein the other devices are devices other than the device among the multiple devices.

[0119] In some embodiments, the first data synchronization module 6551 is also used to perform data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, in response to the latest data of the first parameter group to which the first parameter to be executed belongs not being synchronized to the multiple devices.

[0120] In some embodiments, the device also includes a forward propagation execution module 6554, which is used for executing a third forward propagation corresponding to the first model based on the second parameter and the first forward propagation result corresponding to the first forward propagation when the first parameter group also includes a second parameter to obtain a third forward propagation result.

[0121] In some embodiments, the device also includes a parameter group determination module 6555, which is used to sort the multiple parameters based on the execution order of the multiple parameters to obtain a first parameter sequence; when the parameter quantity of the first parameter sequence is not less than the first data quantity, data extraction is performed on the first parameter sequence starting from the head, the extracted parameters that meet the first data quantity are grouped into a parameter group, and the extracted parameters are removed from the first parameter sequence, wherein the data quantity of the parameter group is the first data quantity; when the parameter quantity of the first parameter sequence is less than the first data quantity, the parameters of the first parameter sequence are grouped into a parameter group, wherein the data quantity of the parameter group is the data quantity of the first parameter sequence.

[0122] In some embodiments, the parameter group determination module 6555 is further used to group the parameters of the first parameter sequence into a first initial parameter group; in response to the data volume of the first initial parameter group not being divisible by a first threshold value, add a virtual parameter to the first initial parameter group so that the first initial parameter group after adding the virtual parameter can be divisible by the first threshold value, and use the first initial parameter group to which the virtual parameter is added as the parameter group.

[0123] In some embodiments, the parameter group determination module 6555 is further configured to group the extracted parameters into a second initial parameter group; in response to the data volume of the second initial parameter group not being divisible by the first threshold, add a dummy parameter to the second initial parameter group so that the second initial parameter group with the added dummy parameter is divisible by the first threshold, and use the second initial parameter group with the added dummy parameter as the parameter group. In some embodiments, the apparatus further includes a parameter allocation module 6556 configured to perform the following processing for each parameter group: evenly distribute the data of all parameters included in the parameter group to the multiple devices.

[0124] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the distributed data processing method described in the embodiment of the present application.

[0125] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the distributed data processing method provided by the embodiment of the present application.

[0126] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0127] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0128] As examples, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). As examples, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected by a communication network.

[0129] In summary, through the embodiment of the present application, data synchronization is performed on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs, wherein the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups; by synchronizing the data of the first parameter group on multiple devices in a targeted manner, on-demand synchronization can be achieved, the amount of data transmission can be reduced, and the synchronization efficiency can be improved. In response to the completion of data synchronization for the first parameter group, the first forward propagation corresponding to the first model is performed based on the first parameter, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is a parameter group whose execution order in the first model is after the first parameter group. After the data synchronization of the first parameter group is completed, the forward propagation based on the first parameter in the first parameter group is immediately started, and the data of the second parameter group is synchronized on multiple devices at the same time, so that the forward propagation calculation and the parameter group synchronization are carried out in parallel, which reduces waiting time and improves overall efficiency. In the parameter execution sequence of the first model, the second parameter group is ranked after the first parameter group. Synchronization of the second parameter group only needs to rely on the latest data of the second parameter group, and forward propagation based on the first parameter in the first parameter group does not need to rely on the second parameter group. Therefore, the parallel processing between data synchronization of the second parameter group and forward propagation based on the first parameter in the first parameter group does not affect the accuracy of the current calculation. This embodiment of the present application achieves the parallel execution of forward propagation and parameter group data synchronization, improving data processing efficiency.

[0130] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A distributed data processing method, characterized in that: The method is applied to a distributed system, the distributed system including a plurality of devices, a plurality of parameters of a first model being divided into a plurality of parameter groups, and for each of the parameter groups, the parameters of the parameter group being divided to the plurality of devices, the method comprising: performing data synchronization on the multiple devices based on the latest data of a first parameter group to which a first parameter to be executed belongs, wherein the first parameter is derived from the multiple parameters, and the first parameter group is derived from the multiple parameter groups; In response to the completion of data synchronization for the first parameter group, a first forward propagation corresponding to the first model is performed based on the first parameters, and data synchronization is performed on the multiple devices based on the latest data of the second parameter group, wherein the second parameter group is a parameter group whose execution order in the first model is after the first parameter group.

2. The method according to claim 1, characterized in that The method further comprises: In response to the completion of data synchronization for the second parameter group, continue to perform data synchronization on the multiple devices based on the latest data of the third parameter group until all parameter groups are synchronized on the multiple devices, wherein the third parameter group is a parameter group whose execution order is after the second parameter group; In response to the completion of the first forward propagation, a second forward propagation corresponding to the first model is performed based on the second parameter group until the forward propagation is completed based on all parameter groups.

3. The method according to claim 2, characterized in that The method further comprises: Determining a training loss of the first model based on a second forward propagation result corresponding to the second forward propagation; Determine a first gradient corresponding to the second parameter group based on the training loss; updating the second parameter group based on the first gradient, and determining a second gradient corresponding to the first parameter group based on the first gradient corresponding to the second parameter group, so as to update the first parameter group based on the second gradient; In response to the second parameter group being updated, performing data synchronization on the plurality of devices based on the updated second parameter group; In response to the completion of updating the first parameter group, data synchronization is performed on the plurality of devices based on the updated first parameter group.

4. The method according to claim 1, wherein The performing data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs includes: Each of the devices performs a process of synchronizing the latest data of the parameters assigned to the device in the first parameter group to other devices, wherein the other devices are devices other than the device among the plurality of devices.

5. The method according to claim 1, wherein The performing data synchronization on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs includes: In response to the latest data of the first parameter group to which the first parameter to be executed belongs not being synchronized to the multiple devices, data synchronization is performed on the multiple devices based on the latest data of the first parameter group to which the first parameter to be executed belongs.

6. The method according to claim 1, characterized in that The first parameter group further includes a second parameter. After performing a first forward propagation corresponding to the first model based on the first parameter, the method further includes: A third forward propagation corresponding to the first model is performed based on the second parameter and the first forward propagation result corresponding to the first forward propagation to obtain a third forward propagation result.

7. The method according to claim 1, characterized in that The plurality of parameters are divided into a plurality of parameter groups, comprising the following steps: sorting the plurality of parameters based on an execution order of the plurality of parameters to obtain a first parameter sequence; When the parameter amount of the first parameter sequence is not less than the first data amount, extracting data from the first parameter sequence starting from the head, grouping the extracted parameters that match the first data amount into a parameter group, and removing the extracted parameters from the first parameter sequence; When the parameter amount of the first parameter sequence is smaller than the first data amount, the parameters of the first parameter sequence are grouped into a parameter group.

8. The method according to claim 7, characterized in that The forming of the parameters of the first parameter sequence into a parameter group comprises: Grouping the parameters of the first parameter sequence into a first initial parameter group; In response to the data amount of the first initial parameter group not being divisible by a first threshold, adding a dummy parameter to the first initial parameter group so that the first initial parameter group after adding the dummy parameter is divisible by the first threshold, and using the first initial parameter group after adding the dummy parameter as the parameter group; The step of grouping the extracted parameters that match the first data volume into a parameter group includes: Combining the extracted parameters into a second initial parameter group; In response to the fact that the amount of data in the second initial parameter group cannot be divided by the first threshold, a virtual parameter is added to the second initial parameter group so that the second initial parameter group after adding the virtual parameter can be divided by the first threshold, and the second initial parameter group with the added virtual parameter is used as the parameter group.

9. The method according to claim 1, characterized in that The parameters of the parameter group are divided into the plurality of devices, comprising the following steps: The data of all parameters included in the parameter group are evenly distributed to the plurality of devices.

10. A processor, characterized in that: The processor is configured to execute the method according to any one of claims 1 to 9.

11. A chip, characterized in that: The chip includes: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing computer-executable instructions or computer programs stored in the memory.

12. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing computer-executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images