Data synchronization method, computing device and computing device cluster

Abstract

The invention discloses a data synchronization method, computing equipment and a computing equipment cluster, and belongs to the technical field of data transmission. The method comprises the following steps: a first calculation unit transmits a synchronization preparation request to a first communication unit before obtaining first data; the first communication unit performs a preparation operation required for a data synchronization operation based on the synchronization preparation request; the first calculation unit transmits a synchronous starting request to the first communication unit after obtaining the first data; the first communication unit sends a notification to a second communication unit of the second computing device based on the synchronization starting request, the notification indicates that the first communication unit can execute a data synchronization operation, and data synchronization needs to be performed between the second computing device and the first computing device; the first communication unit executes the data synchronization operation after the communication units in the plurality of computing devices can execute the data synchronization operation. According to the method and the device, the data synchronization efficiency can be ensured on the basis of shortening the total duration consumed by data synchronization.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a data synchronization method, a computing device, and a computing device cluster. Background Art

[0002] In recent years, the method of training neural networks using training cards has been widely used. At the same time, with the continuous growth of the scale of neural network models and the increase of data sets used for model training, the use of large-scale computing clusters and parallel computing to accelerate training has become a trend. Data parallelism is a common way to achieve parallel computing. In data parallel mode, multiple training cards jointly train the same neural network. Each training card saves a complete model copy of the neural network. The training data set is divided into multiple copies, and each training card trains the neural network based on one of the data subsets. After each training card obtains the first data based on the data set, it is necessary to synchronize data with other training cards based on the first data, so that multiple training cards can update the model parameters of the neural network based on the synchronized data.

[0003] Currently, data synchronization between training cards is achieved through communication operations, and the computing operations and communication operations performed by each training card are executed serially. For example, all computing operations and communication operations share a computing flow, and all computing operations and all communication operations required for training are executed serially.

[0004] However, the total time consumption of computing and communication operations is relatively long, which leads to a longer training time of the neural network. Summary of the invention

[0005] The present application provides a data synchronization method, a computing device, and a computing device cluster. The present application can ensure the efficiency of data synchronization on the basis of shortening the total time consumed by data synchronization. When the present application is used in the training process of a neural network, the training time of the neural network can be optimized. The technical solutions provided by the present application are as follows:

[0006] In the first aspect, the present application provides a data synchronization method. The method is applied to multiple computing devices. The method includes: before obtaining the first data, the first computing unit transmits a synchronization preparation request to the first communication unit, the synchronization preparation request instructs the first communication unit to perform a data synchronization operation based on the first data, the first computing device includes the first computing unit and the first communication unit, and the first computing device is any one of the multiple computing devices; the first communication unit performs the preparation operation required for the data synchronization operation based on the synchronization preparation request; after obtaining the first data, the first computing unit transmits a synchronization start request to the first communication unit, and the synchronization start request instructs the first communication unit to start the data synchronization operation; the first communication unit sends a notification to the second communication unit based on the synchronization start request, and the notification indicates that the first communication unit can perform the data synchronization operation, the second computing device includes the second computing unit and the second communication unit, and the second computing device is any one of the multiple computing devices except the first computing device; the first communication unit performs the data synchronization operation after the communication units in the multiple computing devices are all able to perform the data synchronization operation.

[0007] It can be seen that before the first communication unit performs the data synchronization operation, the first computing unit needs to obtain the first data required for performing the data synchronization operation, and the first communication unit needs to perform the preparation work required for the data synchronization operation. Among them, the synchronization preparation request is sent before the first computing unit obtains the first data, the synchronization start request is sent after the first computing unit obtains the first data, and the first communication unit performs the preparation operation required for the data synchronization operation after obtaining the synchronization preparation request. In this way, the process of the first communication unit performing the preparation operation and the process of the first computing unit obtaining the first data can be performed in parallel, so that the time required to perform the preparation operation and the time for the first computing unit to obtain the first data can overlap at least partly, effectively shortening the total time consumed by the first computing unit to obtain the first data and the first communication unit to perform the preparation operation. In addition, after the first communication unit obtains the synchronization start request, it will send a notification to the second communication unit in the multiple second computing devices to indicate that the first communication unit can perform the data synchronization operation, and after the first communication unit determines that the communication units in the multiple computing devices are all able to perform the data synchronization operation, it performs the data synchronization operation based on the first data. That is, the premise that the communication units of all computing devices in the multiple computing devices perform data synchronization operations also includes: the communication units of all computing devices in the multiple computing devices are able to perform data synchronization operations. After this premise is met, each communication unit performs the data synchronization operation. In this way, it can be ensured that the communication units of all computing devices in the multiple computing devices are in a state where they can perform data synchronization operations before performing the data synchronization operation, thereby ensuring that effective data synchronization can be performed between the communication units in the multiple computing devices, and ensuring the efficiency of data synchronization. Therefore, this method can ensure the efficiency of data synchronization on the basis of shortening the total time consumed by data synchronization.

[0008] When the data synchronization method is applied to the scenario of using multiple training cards to train the same neural network, it can effectively improve the parallelism of model training between multiple training cards, shorten the total time consumed for data synchronization between multiple training cards, and thus shorten the training time of the neural network, optimize the training time of the neural network, and ensure the efficiency of data synchronization between multiple training cards. Similarly, applying the data synchronization method to other scenarios where data synchronization between multiple computing devices is required can also achieve similar effects.

[0009] In one implementation, before the first communication unit sends a notification to the second communication unit based on the synchronization start request, the method further includes: after the first communication unit completes the preparation operation, obtaining the synchronization start request. In this way, the first communication unit does not need to scan the synchronization start request during the preparation operation, and can use more resources to perform the preparation operation, thereby ensuring the efficiency of performing the preparation operation.

[0010] Optionally, the first computing device further includes: a first message storage area. The first computing unit transmits a synchronization preparation request to the first communication unit, including: the first computing unit writes the synchronization preparation request to the first message storage area, and the first communication unit reads the synchronization preparation request from the first message storage area; and / or, the first computing unit transmits a synchronization start request to the first communication unit, including: the first computing unit writes the synchronization start request to the first message storage area, and the first communication unit reads the synchronization start request from the first message storage area.

[0011] When messages are transmitted between the computing unit and the communication unit through the message storage area, since the message storage area has the function of recording messages, the computing unit can continue to perform other operations (such as computing operations) after writing the request to the message storage area, without having to wait for the first communication unit to respond to the message before continuing to perform other operations. This allows the computing unit to continue to perform other operations and the communication unit to respond to the message in parallel, thereby reducing the total time spent on the computing unit continuing to perform other operations and the communication unit responding to the message.

[0012] When performing a data synchronization operation, the first communication unit not only needs to provide data to the second communication unit, but also needs to obtain data from the second communication unit, and provide first synchronization data to the first computing unit based on the obtained data. In one implementation, the first communication unit performs a data synchronization operation, including: the first communication unit operates on the first data so that the second communication unit can obtain the first data; the first communication unit obtains the second data from the second communication unit, and the second data is the data based on which the second communication unit performs the data synchronization operation; the first communication unit provides the first synchronization data to the first computing unit, and the first synchronization data is obtained by the first communication unit based on the second data.

[0013] Optionally, the first synchronization data may be the second data itself, or the result of performing a specified operation on the second data, or the result of performing a specified operation on the second data and the first data. When the first synchronization data is the result of performing a specified operation on the second data, the method further includes: the first communication unit performs the specified operation based on the second data to obtain the first synchronization data. When the first synchronization data is the result of performing a specified operation on the second data, the method further includes: the method further includes: the first communication unit performs the specified operation based on the first data and the second data to obtain the first synchronization data.

[0014] The specified operation may be optionally indicated by the first computing unit. For example, the synchronization preparation request carries operation type information. The operation type information is used to indicate the manner in which the first communication unit obtains the first synchronization data based on the second data. That is, the operation type information indicates which data the first synchronization data is obtained based on, and whether the first synchronization data requires the execution of a specified operation on the data and the type of the specified operation.

[0015] Optionally, the specified operation includes one or more of the following: a mathematical operation and a rearrangement operation. For example, the mathematical operation may be a mapping operation (such as Reduce or AllReduce), a subtraction operation, a minimum operation, a maximum operation, and an average operation. The rearrangement operation may be a concatenation operation, etc.

[0016] In one implementation, the synchronization preparation request carries a destination storage address for storing the first synchronization data. The first communication unit provides the first synchronization data to the first computing unit, including: the first communication unit stores the first synchronization data in the destination storage address, so that the first computing unit obtains the first synchronization data from the destination storage address.

[0017] Similarly, the synchronization preparation request carries a source storage address for storing the first data. Then, before the first communication unit operates on the first data, the method further includes: the first communication unit obtains the first data from the source storage address.

[0018] Optionally, the first computing device further includes: a first shared storage area. The first communication unit and the second communication unit are both capable of reading data stored in the first shared storage area. The first communication unit operates on the first data so that the second communication unit can obtain the first data, including: the first communication unit stores the first data in the first shared storage area so that the second communication unit can read the first data from the first shared storage area.

[0019] Similarly, the second computing device further includes: a second shared storage area. Both the first communication unit and the second communication unit can read data stored in the second shared storage area. Then the first communication unit obtains the second data from the second communication unit respectively, including: the first communication unit reads the second data from the second shared storage area associated with the second communication unit, and the second shared storage area associated with the second communication unit is the shared storage area where the second communication unit writes the second data.

[0020] In one implementation, before the first communication unit performs the data synchronization operation, the method further includes: the first communication unit determines that the communication units in the plurality of computing devices are all capable of performing the data synchronization operation. The implementation process includes: the first communication unit obtains the synchronization start request transmitted by the first computing unit; the first communication unit determines that the second communication unit is capable of performing the data synchronization operation.

[0021] In a second aspect, the present application provides a first computing device, the first computing device comprising: a first computing unit and a first communication unit. The first computing unit is used to transmit a synchronization preparation request to the first communication unit before obtaining the first data, the synchronization preparation request instructing the first communication unit to perform a data synchronization operation based on the first data. The first communication unit is used to perform the preparation operation required for the data synchronization operation based on the synchronization preparation request. The first computing unit is also used to transmit a synchronization start request to the first communication unit after obtaining the first data, the synchronization start request instructing the first communication unit to start the data synchronization operation. The first communication unit is also used to send a notification to the second communication unit based on the synchronization start request, the notification indicating that the first communication unit can perform the data synchronization operation, the second communication unit belongs to the second computing device, and data synchronization needs to be performed between the second computing device and the first computing device. The first communication unit is also used to perform the data synchronization operation after the communication units in multiple computing devices that need to synchronize data with the first computing device are all able to perform the data synchronization operation.

[0022] Optionally, the first communication unit is further configured to obtain a synchronization start request after completing the preparation operation.

[0023] Optionally, the first computing device further includes: a first message storage area. The first computing unit is specifically used to write a synchronization preparation request to the first message storage area, and the first communication unit reads the synchronization preparation request from the first message storage area. And / or, the first computing unit is specifically used to write a synchronization start request to the first message storage area, and the first communication unit reads the synchronization start request from the first message storage area.

[0024] Optionally, the first communication unit performs a data synchronization operation, specifically including: operating on the first data so that the second communication unit can obtain the first data; obtaining second data from the second communication unit, the second data being the data on which the second communication unit performs the data synchronization operation; and providing first synchronization data to the first computing unit, the first synchronization data being obtained by the first communication unit based on the second data.

[0025] Optionally, the first communication unit is further configured to perform a specified operation based on the second data to obtain first synchronization data.

[0026] Optionally, the first communication unit is further configured to perform a specified operation based on the first data and the second data to obtain first synchronization data.

[0027] Optionally, the designated operation is instructed by the first computing unit.

[0028] Optionally, the specified operation includes one or more of the following: a mathematical operation and a rearrangement operation.

[0029] Optionally, the synchronization preparation request carries a destination storage address for storing the first synchronization data. The first communication unit is specifically configured to store the first synchronization data in the destination storage address, so that the first calculation unit obtains the first synchronization data from the destination storage address.

[0030] Optionally, the synchronization preparation request carries a source storage address for storing the first data. The first communication unit is further configured to obtain the first data from the source storage address.

[0031] Optionally, the first computing device further includes: a first shared storage area, and the first communication unit and the second communication unit can both read data stored in the first shared storage area. The first communication unit is specifically used to store the first data in the first shared storage area, so that the second communication unit can read the first data from the first shared storage area.

[0032] Optionally, the second computing device further includes: a second shared storage area, and the first communication unit and the second communication unit can both read data stored in the second shared storage area. Then the first communication unit is specifically used to read the second data from the second shared storage area associated with the second communication unit, and the second shared storage area associated with the second communication unit is the shared storage area where the second communication unit writes the second data.

[0033] Optionally, the first communication unit is further used to obtain a synchronization start request transmitted by the first computing unit; the first communication unit is further used to determine whether the second communication unit is capable of performing a data synchronization operation.

[0034] In a third aspect, the present application provides a computing device including a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method provided in the first aspect of the present application and any possible implementation thereof.

[0035] In a fourth aspect, the present application provides a computing device cluster, including multiple computing devices, the multiple computing devices including multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors running the program instructions, so that the computing device cluster executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0036] In a fifth aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0037] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of an implementation scenario involved in a data synchronization method provided in an embodiment of the present application;

[0039] Figure 2 It is a schematic diagram of a computing unit and a communication unit implemented in a heterogeneous mode provided in an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of a computing unit and a communication unit implemented in an isomorphic mode according to an embodiment of the present application;

[0041] Figure 4 is a schematic diagram of an implementation scenario involved in another data synchronization method provided in an embodiment of the present application;

[0042] Figure 5 is a schematic diagram of an implementation scenario involved in another data synchronization method provided in an embodiment of the present application;

[0043] Figure 6 is a schematic diagram of an implementation scenario involved in another data synchronization method provided in an embodiment of the present application;

[0044] Figure 7 is a flow chart of a data synchronization method provided by an embodiment of the present application;

[0045] Figure 8 is a flowchart of a process in which a first communication unit performs a data synchronization operation provided by an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of a process of synchronizing gradients in a process of jointly training a same neural network using multiple training cards provided in an embodiment of the present application;

[0047] Fig.10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0048] Fig.11 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0050] At present, data parallelism is usually used to accelerate the training process and reasoning process of neural networks. This article takes the acceleration of the training process by data parallelism as an example. In the data parallel mode, multiple training cards jointly train the same neural network. Each training card saves a complete model copy of the neural network. The training data set is divided into multiple data subsets, and each training card trains the neural network based on one of the data subsets. After each training card obtains the first data based on the data subset, it is necessary to synchronize the data with other training cards based on the first data, so that multiple training cards can update the model parameters of the neural network based on the synchronized data. Among them, the training card can be selected as an integrated training card, or other types of computing devices such as computing cards can be used to train the neural network. This article takes the use of training cards to train neural networks as an example.

[0051] The training process is usually divided into a forward calculation phase, a reverse calculation phase, and a gradient update phase. In the forward calculation phase, each training card uses its own data subset to calculate the loss value of the neural network. In the reverse calculation phase, each training card calculates the gradient value based on the loss value obtained by its forward calculation, obtains the gradient value calculated by other training cards, and then obtains the update value used to update the model parameters based on the gradient value calculated by itself and the gradient value calculated by other training cards. For example, after obtaining the gradient value calculated by itself and the gradient value calculated by other training cards, the training card uses the AllReduce operation to accumulate the gradient values ​​calculated by all training cards one by one, and then calculates the average value of the accumulated gradient values, which is the update value used to update the model parameters. When multiple training cards are used to train the same neural network together, each training card obtains the update value according to this logic, which ensures that the update value obtained by each training card is the same. In the parameter update phase, each training card uses the update value (such as the average value of the gradient) to update the model parameters.

[0052] Currently, data synchronization between training cards is achieved through communication operations, and the computing operations and communication operations performed by each training card are executed serially. For example, all computing operations and communication operations share a computing flow, and all computing operations and all communication operations required for training are executed serially.

[0053] However, the total time taken for computing and communication operations is relatively long, resulting in a longer training time for the neural network. Similarly, this problem also exists in other scenarios where data needs to be synchronized between multiple computing devices.

[0054] To this end, an embodiment of the present application provides a data synchronization method. The data synchronization method is applied to multiple computing devices. Each computing device includes: a computing unit and a communication unit. Since the method involves data interaction between multiple computing devices. For ease of description, this article takes a first computing device and a second computing device as an example for illustration. The first computing device is any one of the multiple computing devices included in the multiple computing devices, and the second computing device is any one of the multiple computing devices except the first computing device. The first computing device includes a first computing unit and a first communication unit. The second computing device includes a second computing unit and a second communication unit. In this way, from the perspective of a single computing device, the computing device itself is a first computing device, and the computing devices other than the computing device itself in the multiple computing devices are all second computing devices. Then each computing device is both a first computing device from its own perspective and a second device from the perspective of other computing devices. Therefore, each computing device in the multiple computing devices will perform the operations performed by the first computing device and the second computing device in the text. From the perspective of interaction between computing devices, the interaction between all computing devices in the multiple computing devices will interact in accordance with the interaction mode between the first computing device and the second computing device.

[0055] In the method, before obtaining the first data, the first computing unit transmits a synchronization preparation request to the first communication unit to instruct the first communication unit to perform a data synchronization operation based on the first data. Then, the first communication unit performs the preparation operation required for the data synchronization operation based on the synchronization preparation request. After obtaining the first data, the first computing unit transmits a synchronization start request to the first communication unit to instruct the first communication unit to start the data synchronization operation. Then, based on the synchronization start request, the first communication unit sends a notification to the second communication unit to indicate to the second communication unit that the first communication unit can perform the data synchronization operation. The first communication unit performs the data synchronization operation after the communication units in the multiple computing devices are all able to perform the data synchronization operation.

[0056] It can be seen that before the first communication unit performs the data synchronization operation, the first computing unit needs to obtain the first data required for performing the data synchronization operation, and the first communication unit needs to perform the preparation work required for the data synchronization operation. Among them, the synchronization preparation request is sent before the first computing unit obtains the first data, the synchronization start request is sent after the first computing unit obtains the first data, and the first communication unit performs the preparation operation required for the data synchronization operation after obtaining the synchronization preparation request. In this way, the process of the first communication unit performing the preparation operation and the process of the first computing unit obtaining the first data can be performed in parallel, so that the time required to perform the preparation operation and the time for the first computing unit to obtain the first data can overlap at least partly, effectively shortening the total time consumed by the first computing unit to obtain the first data and the first communication unit to perform the preparation operation. In addition, after the first communication unit obtains the synchronization start request, it will send a notification to the second communication unit in the multiple second computing devices to indicate that the first communication unit can perform the data synchronization operation, and after the first communication unit determines that the communication units in the multiple computing devices are all able to perform the data synchronization operation, it performs the data synchronization operation based on the first data. That is, the premise that the communication units of all computing devices in the multiple computing devices perform data synchronization operations also includes: the communication units of all computing devices in the multiple computing devices are able to perform data synchronization operations. After this premise is met, each communication unit performs the data synchronization operation. In this way, it can be ensured that the communication units of all computing devices in the multiple computing devices are in a state where they can perform data synchronization operations before performing the data synchronization operation, thereby ensuring that effective data synchronization can be performed between the communication units in the multiple computing devices, and ensuring the efficiency of data synchronization. Therefore, this method can ensure the efficiency of data synchronization on the basis of shortening the total time consumed by data synchronization.

[0057] When the data synchronization method is applied to the scenario of using multiple training cards to train the same neural network, it can effectively improve the parallelism of model training between multiple training cards, shorten the total time consumed for data synchronization between multiple training cards, and thus shorten the training time of the neural network, optimize the training time of the neural network, and ensure the efficiency of data synchronization between multiple training cards. Similarly, applying the data synchronization method to other scenarios where data synchronization between multiple computing devices is required can also achieve similar effects.

[0058] In an embodiment of the present application, data synchronization refers to a process in which, when some of the computing devices among multiple computing devices have target data that other computing devices do not have, the target data is shared between the computing devices and the other computing devices, so that the multiple computing devices all have the target data. For example, when multiple training cards are used to jointly train the same neural network, each training card trains the neural network based on a data subset in the training data set to obtain a gradient value corresponding to the data subset, but updating the model parameters of the neural network requires a gradient value based on the entire training data set. Therefore, different training cards need to provide other training cards with their own calculated gradient values, and obtain the gradient values ​​calculated by the other training cards from other training cards, so that each training card can obtain the gradient value calculated based on the entire training data set. This process is a data synchronization process.

[0059] This article provides a detailed introduction to the technical solution of the present application from multiple perspectives, including implementation scenarios, method flow, hardware devices, and software devices.

[0060] The following is an example of an implementation scenario of the embodiment of the present application. Figure 1 Schematic diagram of a data synchronization method according to an embodiment of the present application. Figure 1 As shown, the implementation scenario includes: multiple computing devices 10. Each computing device 10 includes: a computing unit 101 and a communication unit 102. In the same computing device 10, a communication connection is established between the computing unit 101 and the communication unit 102. For example, a communication connection is established between the computing unit 101 and the communication unit 102 via a bus. A communication connection is also established between the communication units 102 in different computing devices 10. For example, a communication connection is established between the communication units 102 of different computing devices 10 via a network. Optionally, the network can be a local area network, the Internet, or other networks, which is not limited in the embodiments of the present application.

[0061] In one implementation, the functions of the computing unit 101 in the data synchronization method provided in the embodiment of the present application can be optionally implemented by software, hardware, firmware, or any combination thereof. For example, the executable program for implementing the functions of the computing unit 101 in the data synchronization method is presented in the form of an application installation package, and after the computing device 10 installs the application installation package, the functions of the computing unit 101 in the data synchronization method can be implemented by running the executable program.

[0062] Similarly, the functions of the communication unit 102 in the data synchronization method provided in the embodiment of the present application can be optionally implemented by software, hardware, firmware, or any combination thereof. For example, the executable program for implementing the functions of the communication unit 102 in the data synchronization method is presented in the form of an application installation package. After the computing device 10 installs the application installation package, the computing device 10 can implement the functions of the communication unit 102 in the data synchronization method by running the executable program.

[0063] In an embodiment of the present application, the computing device 10 is any device having both a computing unit 101 and a communication unit 102. For example, the computing device 10 is a training card, a computing card, a server, a computer, a personal computer, a laptop computer, a mobile phone, a smart phone, a tablet computer, a cloud host, a portable mobile terminal, a multimedia player, an e-book reader, a wearable device, a smart home appliance, an artificial intelligence device, an intelligent wearable device or an intelligent vehicle-mounted device, etc. When the computing device 10 is a training card, the training card may include a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU) or other artificial intelligence (AI) dedicated chips, etc. For example, the training card includes an artificial intelligence core (AICore) and an artificial intelligence central processing unit (AICPU). AICore is optionally a GPU for realizing the function of the computing unit 101, and AICPU is optionally a CPU for realizing the function of the communication unit 102. In addition, multiple training cards can be inserted in the same rack. When the computing device 10 is a server, it can be a server, a server cluster consisting of several servers, or a cloud computing service center. Among them, a large number of basic resources owned by the cloud service provider are deployed in the cloud computing service center. For example, computing resources, storage resources, and network resources are deployed in the cloud computing service center. The cloud computing service center can use this large number of basic resources to implement the functions of the computing device 10 in the data synchronization method provided in the embodiment of the present application.

[0064] For example, it is assumed that data synchronization is required between a first computing device 10 and a plurality of second computing devices 10. The first computing device 10 includes a first computing unit 101 and a first communication unit 102. The second computing device 10 includes a second computing unit 101 and a second communication unit 102. The first computing unit 101 is used to obtain the first data required for the data synchronization operation, and transmit a synchronization preparation request and a synchronization start request to the first communication unit 102, and the synchronization preparation request is transmitted before the first data is obtained, and the synchronization start request is transmitted after the first data is obtained. The second computing unit 101 is used to obtain the second data required for the data synchronization operation, and transmit the synchronization preparation request and the synchronization start request to the second communication unit 102, and the synchronization preparation request is transmitted before the second data is obtained, and the synchronization start request is transmitted after the second data is obtained. And, the first communication unit 102 is used to perform the preparation operation required for the data synchronization operation based on the synchronization preparation request after obtaining the synchronization preparation request, and after obtaining the synchronization start request, send a notification to the second communication unit 102 to notify the second communication unit 102 that the first communication unit 102 can perform the data synchronization operation. The second communication unit 102 is used to perform the preparation operation required for the data synchronization operation based on the synchronization preparation request after obtaining the synchronization preparation request, and send a notification to the first communication unit 102 after obtaining the synchronization start request to notify the first communication unit 102 that the second communication unit 102 can perform the data synchronization operation. The first communication unit 102 and the second communication unit 102 respectively perform the data synchronization operation after both the first communication unit 102 and the second communication unit 102 are able to perform the data synchronization operation.

[0065] By way of example, the method can be applied to AI computing scenarios. For example, the method is applied to a scenario where multiple training cards are used to jointly train the same neural network. In this scenario, the behavior of each training card is consistent. Each training card has a computing unit 101 and a communication unit 102. Each training card is assigned a data subset in the training data set, and each training card stores a complete model copy of the neural network. The computing unit 101 of each training card is used to train the neural network based on the assigned data subset to obtain a gradient value. After the computing unit 101 of any training card obtains the gradient value based on the assigned data subset, it is also necessary to obtain the gradient value obtained by the computing unit 101 of other training cards, and then obtain the update value for updating the model parameters based on the gradient value calculated by itself and the gradient value obtained by the computing unit 101 of other training cards, and then perform the parameter update phase according to the updated value. In this process, since the calculation unit 101 of each training card needs to obtain the gradient value obtained by the calculation unit 101 of other training cards, the calculation unit 101 not only needs to provide the gradient value calculated by itself to the calculation unit 101 of other training cards, but also needs to obtain the gradient value calculated by the calculation unit 101 of other calculation cards. The process of providing gradient values ​​and obtaining gradient values ​​is the process of synchronizing gradient values ​​between multiple training cards. When the process of synchronizing gradient values ​​is implemented by the data synchronization method provided in the embodiment of the present application, the calculation unit 101 of each training card is used to transmit a synchronization preparation request to the communication unit 102 in the same training card before obtaining the gradient value based on the allocated data subset, and after obtaining the gradient value, transmit a synchronization start request to the communication unit 102. The communication unit 102 in each training card is used to perform the preparation operation required for synchronizing the gradient value based on the synchronization preparation request after obtaining the synchronization preparation request, and send a notification to the communication unit 102 in other training cards after obtaining the synchronization start request to notify the communication unit 102 in other training cards that it can synchronize the gradient value. After the communication units 102 in the plurality of training cards are all capable of performing the data synchronization operation, the communication units 102 in the plurality of training cards synchronize the gradient values ​​respectively.

[0066] It should be noted that, although the example uses the data to be synchronized as the calculation result as an example, it does not exclude that the data synchronization method can also synchronize other types of data, and the data synchronization method can synchronize all information represented by data. For example, the data synchronization method provided in the embodiment of the present application can also be applied to a scenario in which multiple computing devices 10 are used to jointly provide a service that needs to guarantee the quality of service (QoS). In this scenario, the QOS of the service can be guaranteed from multiple dimensions, and the multiple dimensions will affect each other. Each computing device 10 is responsible for the guarantee of one dimension respectively. In the process of guaranteeing the QOS of the service, any computing device 10 needs to obtain the execution status of the dimension guaranteed by other computing devices 10, and adjust its own guarantee strategy according to the execution status, so that multiple computing devices 10 can cooperate to guarantee the QOS of the service. The object that needs to be synchronized in this scenario is the execution status represented by data. Among them, the computing unit 101 in the computing device 10 guarantees the work, and the communication unit 102 synchronizes the execution status between different computing devices 10. When there is a need for data synchronization, the computing unit 101 transmits a synchronization preparation request to the communication unit 102 in the same computing device 10, then summarizes the execution status that needs to be synchronized, and after obtaining the execution status, transmits a synchronization start request to the communication unit 102. The communication unit 102 is used to perform the preparation operation required for synchronizing the execution status based on the synchronization preparation request after obtaining the synchronization preparation request, and after obtaining the synchronization start request, send a notification to the communication unit 102 in other computing devices 10 to notify the communication unit 102 in other computing devices 10 that it can synchronize the execution status. After the communication units 102 in the multiple computing devices 10 are all able to perform the data synchronization operation, the communication units 102 in the multiple computing devices 10 synchronize the execution status respectively.

[0067] Similarly, the data synchronization method provided in the embodiment of the present application can also be applied to other scenarios that require synchronizing data between multiple computing devices 10, and as business needs change, the synchronization objects can also change with the application scenarios, which are not listed one by one in the embodiment of the present application.

[0068] Optionally, the computing device 10 may include at least one computing unit 101 and at least one communication unit 102. The computing units 101 and the communication units 102 in the same computing device 10 may be in a one-to-one correspondence or a many-to-one relationship. When the computing unit 101 and the communication unit 102 are in a one-to-one correspondence, the communication unit 102 is used to synchronize data for the corresponding computing unit 101. When the computing unit 101 and the communication unit 102 are in a many-to-one relationship, one communication unit 102 is used to synchronize data for multiple corresponding computing units 101.

[0069] In the present application, the functions of the computing unit 101 and the communication unit 102 can be optionally turned on and off according to the needs, and the timing of turning on and off can be adjusted according to the application needs. For example, since the computing unit 101 is mainly used to perform computing operations, the function of the computing unit 101 can be optionally turned on when there is a computing demand and turned off when there is no computing demand, which can save resource overhead. The communication unit 102 is mainly used to synchronize data for the computing unit 101. When the computing unit 101 has a data synchronization demand, it is sufficient to ensure that the function of the communication unit 102 is in an on state. In one implementation, the function of the communication unit 102 can be optionally turned on at the same time when the function of the computing unit 101 is turned on. For example, when the function of the computing unit 101 is turned on, the function of the communication unit 102 is turned on at the same time. In another implementation, the function of the communication unit 102 can be optionally kept in a normally open state. For example, when the computing unit 101 and the communication unit 102 are in a many-to-one relationship, the function of the communication unit 102 can be optionally kept in a normally open state. In this way, when the function of any computing unit 101 corresponding to the communication unit 102 is in the on state, the communication unit 102 can synchronize data for the computing unit 101. And since there is no need to switch the switch state with the functional states of multiple computing units 101, frequent switching of the functional state of the communication unit 102 can be avoided.

[0070] In some implementation scenarios, when the function of the communication unit 102 remains in a normally open state, the function of the communication unit 102 can be optionally provided in the form of a service. When the computing unit 101 needs the communication unit 102 to synchronize data, the computing unit 101 can request the communication unit 102 to perform a data synchronization operation for the computing unit 101 by requesting a service. For example, the computing unit 101 transmits a synchronization preparation request to the communication unit 102, which can be regarded as the computing unit 101 requesting a data synchronization service. The communication unit 102 performs the preparation operation required for the data synchronization operation based on the synchronization preparation request, which can be regarded as the communication service issuing a task to prepare for the data synchronization service based on the synchronization preparation request. The computing unit 101 transmits a synchronization start request to the communication unit 102, which can be regarded as the computing unit 101 notifying the communication unit 102 to start providing the data synchronization service. The communication unit 102 performs the data synchronization operation based on the synchronization start request, which can be regarded as the communication service providing the data synchronization service based on the synchronization start request. After the data synchronization operation is completed, the data synchronization service process is completed.

[0071] In addition, the functions of the computing unit 101 and the communication unit 102 can also be implemented by operators. That is, the functions of the computing unit 101 are implemented by computing operators, and the functions of the communication unit 102 are implemented by communication operators. At this time, the functions of the computing unit 101 and the communication unit 102 can be enabled by pulling up the corresponding operators. Among them, a collection of one or more operations performed on an operation object is called an operator.

[0072] In the present application, there are multiple presentation modes of the computing unit 101 and the communication unit 102. The following is an example of the computing unit 101 and the communication unit 102 being presented in a homogeneous mode and a heterogeneous mode.

[0073] In a first implementation, the computing unit 101 and the communication unit 102 are implemented in a heterogeneous mode. In this implementation, different hardware that differs in hardware structure exists in the computing device 10, and the computing unit 101 and the communication unit 102 are respectively implemented by different hardware. For example, the computing device 10 includes a first hardware and a second hardware, and the first hardware and the second hardware differ in hardware structure. The first hardware can implement the function of the computing unit 101 by running an executable program for implementing the function of the computing unit 101. The second hardware can implement the function of the communication unit 102 by running an executable program for implementing the function of the communication unit 102. Since the hardware used to implement the function of the computing unit 101 and the hardware used to implement the function of the communication unit 102 differ in hardware structure, the computing unit 101 and the communication unit 102 can be regarded as implemented in a heterogeneous mode. For example, Figure 2 As shown, the first hardware can be selected as AICore, and the second hardware can be selected as AICPU.

[0074] In a second implementation, the computing unit 101 and the communication unit 102 are implemented in an isomorphic mode. In this implementation, the functions of the communication unit 102 and the computing unit 101 can be implemented by hardware having the same hardware structure. For example, when the computing device 10 has multiple first hardware, some of the first hardware can implement the functions of the computing unit 101 by running an executable program for implementing the functions of the computing unit 101, and some of the first hardware can implement the functions of the communication unit 102 by running an executable program for implementing the functions of the communication unit 102. Since the hardware used to implement the functions of the computing unit 101 and the hardware used to implement the functions of the communication unit 102 in this implementation are the same in hardware structure, the computing unit 101 and the communication unit 102 can be regarded as implemented in an isomorphic mode. For example, as Figure 3 As shown, the computing device 10 includes multiple AICores, some of which are used to implement the functions of the computing unit 101, and other AICores are used to implement the functions of the communication unit 102.

[0075] Alternatively, if Figure 4 As shown, each computing device 10 also includes a message storage area 103. The message storage area 103 is used to transfer messages between the computing unit 101 and the communication unit 102 of the same computing device 10. For example, by the first computing unit 101 writing a synchronization preparation request to the first message storage area 103, and the first communication unit 102 reading the synchronization preparation request from the first message storage area 103, the synchronization preparation request can be transferred from the first computing unit 101 to the first communication unit 102. And / or, by the first computing unit 101 writing a synchronization start request to the first message storage area 103, and the first communication unit 102 reading the synchronization start request from the first message storage area 103, the synchronization start request can be transferred from the first computing unit 101 to the first communication unit 102. In one implementation, the message storage area 103 is a memory. When the computing unit 101 and the communication unit 102 transmit messages through the message storage area 103, since the message storage area 103 has the function of recording messages, the computing unit 101 can continue to perform other operations (such as computing operations) after writing the request to the message storage area 103, without waiting for the first communication unit 102 to respond to the message before continuing to perform other operations, so that the process of the computing unit 101 continuing to perform other operations and the process of the communication unit 102 responding to the message can be performed in parallel, thereby reducing the total time consumed by the process of the computing unit 101 continuing to perform other operations and the process of the communication unit 102 responding to the message. Among them, the corresponding relationship between the computing unit 101 and the message storage area 103, and the communication unit 102 and the message storage area 103 can be configured according to application requirements. For example, the computing unit 101 and the message storage area 103, and the communication unit 102 and the message storage area 103 can be selected as a one-to-one relationship, a many-to-one relationship, or a one-to-many relationship. The message storage area 103 is used to transmit messages to its corresponding computing unit 101. Among them, if Figure 5 As shown, each computing device includes n computing units, n message storage areas 103 and 1 communication unit. It can be seen that the computing unit and the message storage area 103 are in a one-to-one correspondence, and the communication unit and the message storage area 103 are in a many-to-one relationship. Figure 6 As shown, each computing device includes 1 computing unit, 1 message storage area 103 and 1 communication unit. It can be seen that the computing unit and the message storage area 103 are in a one-to-one correspondence, and the communication unit and the message storage area 103 are in a one-to-one correspondence.

[0076] Furthermore, if Figure 4As shown, each computing device 10 also includes a shared storage area 104. The communication units 102 of different computing devices 10 exchange data through the shared storage area 104. For example, the first computing device 10 also includes a first shared storage area 104. The second computing device 10 also includes a second shared storage area 104. After the first communication unit 102 stores the first data in the first shared storage area 104, the second communication unit 102 can read the first data from the first shared storage area 104 to synchronize the first data to the second communication unit 102. Correspondingly, after the second communication unit 102 stores the second data in the second shared storage area 104, the first communication unit 102 can read the second data from the second shared storage area 104 to synchronize the second data to the first communication unit 102. In one implementation, the shared storage area 104 is a memory. The corresponding relationship between the communication unit 102 and the shared storage area 104 can be configured according to application requirements. For example, the communication unit 102 and the shared storage area 104 can be selected as a one-to-one corresponding relationship, a one-to-many relationship, or a many-to-one relationship. The shared storage area 104 is used to store data based on which data synchronization operations are performed for its corresponding communication unit 102. The data based on which data synchronization operations are performed refers to the data that needs to be provided to the communication unit 102 in other computing devices to perform data synchronization operations. And when a communication unit 102 corresponds to multiple shared storage areas 104, the communication unit 102 can synchronize data with communication units in other computing devices 10 through multiple shared storage areas 104. When multiple communication units 102 correspond to one shared storage area 104, the multiple communication units 102 synchronize data by time-division multiplexing the shared storage area 104. Among them, if Figure 5 As shown, each computing device includes a shared storage area 104 and a communication unit. It can be seen that the communication unit and the shared storage area 104 are in a one-to-one correspondence. Figure 6 As shown, each computing device includes a shared memory area 104 and a communication unit, and it can be seen that there is a one-to-one correspondence between the communication unit and the shared memory area 104. Since the shared memory area 104 is used to provide data between the communication units of different computing devices, the shared memory area 104 is also called a window area.

[0077] like Figure 4As shown, each computing device 10 also includes a data storage area 105. Data that needs to be transmitted between the computing unit 101 and the communication unit 102 in the same computing device 10 can be transmitted through the data storage area 105. For example, after the computing unit 101 stores the data in the data storage area 105, the communication unit 102 can obtain the data by reading the data from the data storage area 105, and perform data synchronization based on the data. Similarly, after the communication unit 102 obtains data from other communication units 102, it can store the synchronized data in the data storage area 105, and the computing unit 101 can obtain the synchronized data by reading the synchronized data from the data storage area 105. Furthermore, when the computing device 10 has multiple computing units 101, the computing device 10 has multiple data storage areas 105 corresponding to the multiple computing units 101, respectively, or the multiple computing units 101 share one data storage area 105. In one implementation, the data storage area 105 is a memory. By way of example, the data storage area 105 can be a global memory (GM). Among them, as Figure 5 As shown, each computing device includes n computing units, 1 data storage area 105 and 1 communication unit. It can be seen that the computing unit and the data storage area 105 are in a many-to-one relationship, and the communication unit and the data storage area 105 are in a one-to-one relationship. Figure 6 As shown, each computing device includes 1 computing unit, 1 data storage area 105 and 1 communication unit. It can be seen that the computing unit and the data storage area 105 are in a one-to-one correspondence, and the communication unit and the data storage area 105 are in a one-to-one correspondence.

[0078] It should be noted that when any two of the message storage area 103, the shared storage area 104 and the data storage area 105 are storage media with the same structure, any two of them can be storage media that are independent of each other in physical structure, or they can be different storage areas in the same storage medium, and the embodiments of the present application do not make specific limitations on this.

[0079] It should be understood that the above content is an illustrative description of the implementation scenario of the data synchronization method provided in the embodiment of the present application, and does not constitute a limitation on the implementation scenario of the data synchronization method. A person of ordinary skill in the art can know that as business needs change, the implementation scenario can be adjusted according to application requirements, and the embodiment of the present application does not list them one by one.

[0080] The data synchronization method provided by the embodiment of the present application is described below. Figure 7 As shown, the data synchronization method includes the following steps:

[0081] Step 701: Before obtaining first data, the first computing unit transmits a synchronization preparation request to the first communication unit, where the synchronization preparation request instructs the first communication unit to perform a data synchronization operation based on the first data.

[0082] The first computing unit may optionally directly transmit the synchronization preparation request to the first communication unit. For example, the first computing unit sends the synchronization preparation request to the first communication unit. Alternatively, the first computing unit indirectly transmits the synchronization preparation request to the first communication unit. For example, when the first computing device has a first message storage area, the first computing unit writes the synchronization preparation request to the first message storage area, and after the first communication unit reads the synchronization preparation request from the first message storage area, the synchronization preparation request can be transmitted from the first computing unit to the first communication unit.

[0083] As an implementation of the synchronization preparation request, the synchronization preparation request carries a source storage address and a destination storage address. The source storage address is used to store the first data. The destination storage address is used to store the first synchronization data. For example, the source storage address and the destination storage address are both addresses in a data storage area (such as a memory). For example, when 10 training cards are used to train a neural network together, the synchronization preparation request sent by the computing unit in any training card carries the address of the first buffer (such as the address of srcBuffer) and the address of the second buffer (such as the address of receiveBuffer) in the training card. The first buffer is used to store the first data. The second buffer is used to store the first synchronization data. Among them, the source storage address can be optionally represented by a starting address and a length. The destination storage address can be optionally represented by a starting address, or by a starting address and a length (such as bufferlen). For example, when the length of the data synchronized between multiple computing devices is consistent, the length of the first synchronization data can be determined based on the length of the first data, and the destination storage address is represented by the starting address and the length. When the synchronization preparation request carries the source storage address and the destination storage address, no matter what the correspondence between the computing unit and the communication unit in the same computing device is, such as one-to-one correspondence or many-to-one correspondence, and no matter what the correspondence between the computing unit and the message storage area in the computing device is, such as one-to-one correspondence, many-to-one or one-to-many correspondence, after the first communication unit obtains the synchronization preparation request transmitted by any first computing unit, it can determine the storage address of the first data and the first synchronization data involved in the synchronization preparation request based on the storage address carried by the synchronization preparation request, and interact with the first computing unit based on the address, without being confused with the first data and the first synchronization data involved in other synchronization preparation requests, and can ensure the correct transmission of data during the data synchronization operation. It should be noted here that the synchronization preparation request indicates that a data synchronization operation is performed based on the first data, which does not mean that the synchronization preparation request carries the first data, but indicates the first data by carrying the source storage address, so as to indicate that a data synchronization operation is performed based on the first data.

[0084] Optionally, the synchronization preparation request may also optionally carry operation type information (optype). The operation type information is used to indicate the manner in which the first communication unit obtains the first synchronization data based on the second data. The second data is the data obtained by the first communication unit from the second communication unit by performing a data synchronization operation. For example, the operation type information indicates that the first synchronization data is the second data itself, or is the result of performing a specified operation on the second data, or is the result of performing a specified operation on the second data and the first data. That is, the operation type information indicates which data the first synchronization data is obtained based on, and whether the first synchronization data needs to perform a specified operation on the data and the type of the specified operation. In one implementation, the specified operation includes one or more of the following: mathematical operations and rearrangement operations. For example, the mathematical operation may be a mapping operation (Reduce or AllReduce), an addition operation (Add), a subtraction operation (Subtract), a minimum operation (Min), a maximum operation (Max), and an average operation (Average). Reduce is to map the data in multiple processes according to a specified mapping function, and store the mapping results in one process. AllReduce is to map the data in multiple processes according to the specified mapping function, and store the mapping results in the multiple processes. The rearrangement operation can be a splicing operation (Gather or AllGather), etc. Gather is to splice the data of multiple processes together and store the splicing results in one process. AllGather is to splice the data of multiple processes together and store the splicing results in the multiple processes. For example, taking the use of 10 training cards to train a neural network as an example, after the communication unit in each training card obtains the gradient value calculated by the computing unit in the remaining 9 training cards, the gradient values ​​of the 10 training cards can be accumulated one by one through the AllReduce operation, and then the average value of the accumulated gradient value is calculated, and the average value is synchronized to the computing unit associated with the communication unit. The computing unit associated with the communication unit is a computing unit that provides a synchronization preparation request to the communication unit. It should be noted that the way in which the first communication unit obtains the first synchronization data based on the second data is indicated by the first computing unit, and the indication by the synchronization preparation request is only an example, and it can also have other implementation methods. For example, this method can be a default operation mode configured to the first communication unit, and the embodiment of the present application does not specifically limit it.

[0085] Considering that it takes a certain period of time for the first computing unit to obtain the first data after it starts to obtain the first data, there are at least three implementation methods for the first computing unit to send the synchronization preparation request:

[0086] In a first implementation, the first computing unit sends a transmission preparation request to the first communication unit before starting to obtain the first data. For example, the first computing unit first transmits a synchronization preparation request to the first communication unit, and then starts to obtain the first data. Optionally, transmitting the synchronization preparation request and obtaining the first data can be performed serially.

[0087] In a second implementation, the first computing unit transmits a synchronization preparation request to the first communication unit when starting to obtain the first data. For example, the first computing unit configures two parallel threads, one of the two threads is used to obtain the first data, and the other thread is used to transmit the synchronization preparation request to the first communication unit, and the two threads start executing at the same time.

[0088] In a third implementation, the first computing unit transmits a synchronization preparation request to the first communication unit in the intermediate period from the start of acquiring the first data to the acquisition of the first data. For example, the first computing unit is configured with two parallel threads, one of which is used to acquire the first data and the other is used to transmit the synchronization preparation request to the first communication unit, and the first computing unit immediately executes the thread that transmits the synchronization preparation request after the thread that acquires the first data starts to execute.

[0089] It should be noted that an initialization process needs to be completed before executing step 701. For example, the initialization process includes: creating a communication domain including the multiple computing devices, configuring a message storage area and a data storage area in each computing device, and reserving memory for implementing a shared storage area.

[0090] Step 702: The first communication unit performs a preparation operation required for a data synchronization operation based on the synchronization preparation request.

[0091] After obtaining the synchronization preparation request, the first communication unit can determine that the computing unit has a data synchronization requirement. In order to perform the data synchronization operation, the first communication unit needs to first perform the preparation operation required for the data synchronization operation. For example, the synchronization preparation request is a "prepare" notification. When the first computing unit has a data synchronization requirement, the "prepare" notification is sent to the first communication unit. After the first communication unit obtains the "prepare" notification, it can determine that the first computing unit has a data synchronization requirement, and then perform the preparation operation required for the data synchronization operation. In one implementation, performing the preparation operation includes: the first communication unit and the second communication unit negotiate a communication algorithm used for communication between the two, allocate a shared storage area in the first computing device and the second computing device, and align the clocks of the first communication unit and the second communication unit. Among them, when the synchronization preparation request carries the source storage address of the first data, the first communication unit can determine the size of the first data based on the source storage address, and estimate the size of the shared data area required to perform the data synchronization operation based on the first data, so that the first computing device allocates the shared data area in the first computing device according to the size. Similarly, the second communication unit can also allocate the shared data area in the second computing device in this way. For example, assuming that the first communication unit can obtain the total number of computing devices that need to synchronize data, and the size of the data based on which the communication units in the multiple computing devices perform data synchronization operations is the same, after the first communication unit determines the size of the first data, the size of the required shared data area can be obtained based on the size and the total number. For example, the size of the shared data area is equal to the product of the size of the first data and the total number.

[0092] Step 703: After obtaining the first data, the first computing unit transmits a synchronization start request to the first communication unit, where the synchronization start request instructs the first communication unit to start a data synchronization operation.

[0093] The first computing unit may optionally directly transmit the synchronization start request to the first communication unit. For example, the first computing unit sends the synchronization start request to the first communication unit. Alternatively, the first computing unit indirectly transmits the synchronization start request to the first communication unit. For example, when the first computing device has a first message storage area, the first computing unit writes the synchronization start request to the first message storage area, and after the first communication unit reads the synchronization start request from the first message storage area, the synchronization start request can be transmitted from the first computing unit to the first communication unit.

[0094] According to the foregoing content, it can be known that the first communication unit needs to respond to the paired synchronization preparation request and the synchronization start request to perform the data synchronization operation. Therefore, the synchronization start request and the synchronization preparation request may optionally carry information indicating that the two are paired synchronization start request and synchronization preparation request. For example, the synchronization start request carries associated information, and the associated information indicates the synchronization preparation request that is paired with the synchronization start request. For example, the synchronization preparation request carries a request number, and the associated information of the synchronization start request indicates that the synchronization start request is paired with the synchronization preparation request identified by the request number. For another example, the synchronization preparation request and the synchronization start request both carry a source storage address, and the synchronization preparation request and the synchronization start request carrying the same source storage address are paired. Among them, the paired synchronization preparation request and the synchronization start request indicate the same data synchronization operation.

[0095] Step 704: After completing the preparation operation, the first communication unit obtains a synchronization start request.

[0096] In the present application, the execution sequence of the first communication unit performing the preparation operation and obtaining the synchronization start request can have at least the following two types. The first execution sequence: after completing the preparation operation, the first communication unit scans the synchronization start request again, and after scanning the synchronization start request, obtains the synchronization start request. The second execution sequence: the first communication unit scans the synchronization start request during the preparation operation, and after scanning the synchronization start request, obtains the synchronization start request. In the first execution sequence, the first communication unit does not need to scan the synchronization start request during the preparation operation, and can use more resources to perform the preparation operation to ensure the efficiency of the preparation operation. In the second execution sequence, the first communication unit also needs to scan the synchronization start request during the preparation operation. The operation of scanning the synchronization start request will affect the efficiency of the preparation operation and increase the overhead. In addition, since the synchronization preparation request is sent before the synchronization start request, if the synchronization start request is scanned after the preparation operation is completed, the logic of the first communication unit consuming the request can be simplified, which is easier to implement. Among them, the first communication unit can optionally scan the synchronization start request periodically. For example, when the first computing unit transmits a request to the second communication unit through the message storage area, the first communication unit periodically scans the message storage area, and when a new request is found in the message storage area, the first communication unit obtains the request from the message storage area.

[0097] Step 705: The first communication unit sends a notification to the second communication unit based on the synchronization start request, where the notification indicates that the first communication unit can perform a data synchronization operation.

[0098] After the first communication unit obtains the synchronization start request, in order to ensure that the communication units in multiple computing devices can perform effective data synchronization, it is necessary to send a notification to each second communication unit to notify the first communication unit that it is in a state where it can perform data synchronization operations, so that the second communication unit can determine whether the communication units of all computing devices are in a state where they can perform data synchronization operations, and determine whether to perform data synchronization operations based on the judgment result. For example, after the first communication unit obtains the synchronization start request, it can be known that the first computing unit has completed the preparation work for the first data, and the first communication unit determines that it can perform data synchronization operations. For example, the synchronization start request is a "work" notification. After the first computing unit obtains the first data, it sends a "work" notification to the first communication unit. After the first communication unit obtains the "work" notification, the first communication unit determines that it can perform data synchronization operations, and then sends a "ready" notification to the second communication unit to notify the second communication unit that the first communication unit can perform data synchronization operations.

[0099] Among them, when the first communication unit is in a state capable of performing data synchronization operations, the first communication unit itself needs to meet at least two conditions, the first condition is that the first computing unit has completed the preparation of the first data, and the second condition is that the first communication unit has completed the preparation operations required for performing data synchronization operations. After the first computing unit completes the preparation of the first data, it will send a synchronization start request to the first communication unit. After the first communication unit receives the synchronization start request, it can determine that the first computing unit has completed the preparation of the first data. Among them, the first computing unit completes the preparation of the first data mainly including: the first computing unit obtains the first data, and the first data has been stored at the source storage address. In most scenarios, such as the scenario where the first computing unit obtains the first data through a calculation operation, since the time consumed to store the first data in the source storage address is negligible compared to the time taken by the first computing unit to perform the calculation operation, the first condition can be regarded as the first computing unit obtaining the first data.

[0100] Optionally, considering that multiple computing devices may execute multiple data synchronization processes in the same period, in order to ensure that the communication unit can identify notifications belonging to the same data synchronization process, the notification may optionally carry information identifying the data synchronization process. The data synchronized in the same data synchronization process is data with correlation. For example, when multiple training cards are used to train a neural network, the data synchronized in the same data synchronization process is the gradient value calculated by each training card based on the assigned data subset in the same iteration process.

[0101] Step 706: After the communication units in the plurality of computing devices are all capable of performing the data synchronization operation, the first communication unit performs the data synchronization operation.

[0102] Before performing a data synchronization operation, the first communication unit needs to determine whether the communication units in multiple computing devices are all capable of performing the data synchronization operation. The first communication unit determines that the communication units in multiple computing devices are all capable of performing the data synchronization operation when the first communication unit has obtained the synchronization start request transmitted by the first computing unit and all the second communication units in the multiple computing devices are all capable of performing the data synchronization operation. In one implementation, since the first communication unit needs to send a notification to each second communication unit after obtaining the synchronization start request, the first communication unit will also receive the notification sent by the second communication unit and count the received notifications, such as adding one to the count after each notification is received. When the total number of notifications received is equal to the total number of second communication units that need to perform data synchronization, it is determined that the second communication units in multiple computing devices are all capable of performing the data synchronization operation. When the notification carries information identifying the data synchronization process, the first communication unit may optionally count each data synchronization process separately. After receiving each notification, it first obtains the information identifying the data synchronization process, and then increases the count of the data synchronization process indicated by the information by one. When the total number of any data synchronization process is equal to the total number of second communication units participating in the data synchronization process, it is determined that all second communication units participating in the data synchronization process are able to perform the data synchronization operation.

[0103] According to the previous description, when the first communication unit performs a data synchronization operation, it not only needs to provide data to the second communication unit, but also needs to obtain data from the second communication unit, and provide the first synchronization data to the first computing unit based on the obtained data. For example, when 10 training cards are used to train a neural network, the communication unit in any training card needs to provide the gradient value calculated by the computing unit in the training card to the communication units in other training cards, and also needs to obtain the gradient value calculated by the computing units in the remaining 9 training cards, and then provide the first synchronization data to the first computing unit based on the gradient value calculated by the computing units in the 10 training cards. Then in one implementation, Figure 8 As shown, the process of the first communication unit performing the data synchronization operation includes:

[0104] Step 7061: The first communication unit operates on the first data so that the second communication unit can obtain the first data.

[0105] When the synchronization preparation request carries the source storage address of the first data, the process of the first communication unit executing step 7061 includes: the communication unit obtains the first data from the source storage address, and operates on the first data so that the second communication unit can obtain the first data. In one implementation, when the first computing device includes a first shared storage area, since both the first communication unit and the second communication unit can read the data stored in the first shared storage area, the first communication unit operates on the first data so that at least one second communication unit can obtain the first data, including: the first communication unit stores the first data in the first shared storage area, so that at least one second communication unit can read the first data from the first shared storage area. Alternatively, the first communication unit may also optionally adopt other methods to enable the second communication unit to obtain the first data, for example, the first communication unit broadcasts the first data to the second communication unit, etc., which is not specifically limited in the embodiments of the present application.

[0106] Optionally, the first communication unit may also operate on the first data including: adding information identifying the data synchronization process to the first data, so that the second communication unit can identify the data synchronized in the same data synchronization process to ensure that the required data can be synchronized. And / or, the first communication unit may also add the identification information of the first communication unit to the first data, so that the second communication unit can determine that the first data is the data that the first communication unit needs to synchronize based on the identification information. It should be noted that if there are other solutions that can ensure the above identification results, the first communication unit may not add corresponding information to the first data. For example, if the communication units of a computing device in the same time period are only allowed to participate in one data synchronization process, when the second communication unit synchronizes data with the first communication unit, the data obtained by the second communication unit in the shared storage area corresponding to the first communication unit can ensure that it is the data synchronized by the first communication unit to the second communication unit. At this time, there is no need to add information to the first data to ensure the identification result.

[0107] Step 7062: The first communication unit obtains second data from the second communication unit, where the second data is data based on which the second communication unit performs a data synchronization operation.

[0108] When the second computing device includes a second shared storage area, since both the first communication unit and the second communication unit can read the data stored in the second shared storage area, the first communication unit obtains the second data from the second communication unit, including: the first communication unit reads the second data from the second shared storage area associated with the second communication unit. The second shared storage area associated with the second communication unit is a shared storage area to which the second communication unit writes the second data. For example, when three training cards are used to train a neural network, training card 1 includes communication unit 1 and shared storage area 1, training card 2 includes communication unit 2 and shared storage area 2, and training card 3 includes communication unit 3 and shared storage area 3. Data synchronization is required between communication unit 1, communication unit 2, and communication unit 3. Then, communication unit 1 writes data 1 that needs to be provided to communication units 2 and 3 to shared storage area 1, communication unit 2 writes data 2 that needs to be provided to communication units 1 and 3 to shared storage area 2, and communication unit 3 writes data 3 that needs to be provided to communication units 1 and 2 to shared storage area 3. Communication unit 1 needs to read data 2 from shared storage area 2 and read data 3 from shared storage area 3. Communication unit 2 needs to read data 1 from shared memory area 1 and data 3 from shared memory area 3. Communication unit 3 needs to read data 2 from shared memory area 2 and data 1 from shared memory area 1.

[0109] Step 7063: The first communication unit provides first synchronization data to the first computing unit, where the first synchronization data is obtained by the first communication unit based on the second data.

[0110] After the first communication unit obtains the second data, it can provide the first synchronization data to the first computing unit based on the second data. The first synchronization data is obtained based on the second data. The first computing unit can indicate to the first communication unit the way to obtain the first synchronization data based on the second data. For example, the synchronization preparation request carries operation type information, and the operation type information indicates the way. Then, after the first communication unit obtains the second data, the first synchronization data can be obtained based on the second data in the way indicated by the operation type information. For example, when the operation type indication information indicates that the first synchronization data is the second data itself, the first communication unit can provide the second data to the first computing unit. When the operation type indication information indicates that the first synchronization data is the result of performing a specified operation on the second data, the first communication unit can perform the specified operation on the second data to obtain the first synchronization data, and provide the first synchronization data to the first computing unit. When the operation type indication information indicates that the first synchronization data is the result of performing a specified operation on the second data and the first data, the first communication unit can perform the specified operation on the second data and the first data to obtain the first synchronization data, and provide the first synchronization data to the first computing unit.

[0111] When the synchronization preparation request carries a destination storage address of first synchronization data, the first communication unit provides the first synchronization data to the first calculation unit, including: the first communication unit stores the first synchronization data in the destination storage address, so that the first calculation unit obtains the first synchronization data from the destination storage address.

[0112] The following uses the example of using multiple training cards to jointly train the same neural network to illustrate the implementation process of the data synchronization method. Assume that the data synchronization method is used to synchronize the gradients calculated by the computing units in N training cards, where N is a positive integer. The computing unit is the AICore in the training card, and the communication unit is the AICpu in the training card. The function of the communication unit is presented in the form of a data synchronization service. Then the data synchronization implemented by AICore and AICpu includes the following stages:

[0113] Phase 1: Initialization phase. It includes the following steps:

[0114] 1. After all N training cards are started, the N training cards jointly create a communication domain including the N training cards, configure a message storage area and a data storage area in each training card, and reserve memory for implementing a shared storage area, etc.

[0115] Phase 2: AICpu prepares for data synchronization services. Fig. 9 As shown, it includes the following steps:

[0116] 1. Before AICore calculates the gradient based on the assigned data subset, it writes a "prepare" notification to the message storage area in the training card where it is located. When AICpu scans a new message in the message storage area in its training card, it obtains the "prepare" notification from the message storage area. The "prepare" notification carries the address of srcBuffer, the address of receiveBuffer, optype and bufferlen. srcBuffer is used to store the gradient of this calculation. ReceiveBuffer is used to store the first synchronization data obtained based on the gradient calculated by N training cards. At this point, it can be regarded as AICore requesting data synchronization service from AICpu through the "prepare" notification.

[0117] 2. After receiving the "prepare" notification, the AICpu performs the preparation operations required for data synchronization. For example, it negotiates the communication algorithm with the AICpu in other training cards, allocates shared storage areas in the training card, and aligns the clock with the AICpu in other N-1 training cards. At the same time, AICore performs calculation operations based on the allocated data subset to obtain the gradient.

[0118] The third stage: AICpu provides data synchronization services. Fig. 9 As shown, it includes the following steps:

[0119] 1. After AICpu completes the preparation operation, it continues to scan the messages in the message storage area of ​​the training card where it is located to wait for the message from AICore.

[0120] 2. After AICore calculates the gradient based on the assigned data subset, it writes a "work" notification to the message storage area in the training card where it is located. When AICpu scans a new message in the message storage area in the training card where it is located, it obtains the "work" notification from the message storage area. At this point, it can be considered that AICore instructs AICpu to start providing data synchronization services through the "work" notification.

[0121] 3. After receiving the "work" notification, the AICpu sends a "ready" notification to the AICpu in the other N-1 training cards, indicating that the AICpu in the current training card can perform data synchronization operations.

[0122] 4. After determining that the AICpu in the N training cards are all capable of performing data synchronization operations, the AICpu performs data synchronization operations. For example, according to the address of srcBuffer and bufferlen, the gradient calculated by the AICore is obtained from the data storage area, and the gradient is stored in the shared storage area of ​​the current training card, and the gradient is obtained from the shared storage area of ​​the other N-1 training cards, and then the first synchronization data is obtained based on all the obtained gradients in the manner indicated by optype, and the first synchronization data is stored in the address of receiveBuffer, so that the AICore obtains the first synchronization data from the address of receiveBuffer.

[0123] 5. After AICpu stores the first synchronization data in the address of receiveBuffer, it sends an "end" notification to the AICore in the corresponding training card to notify AICore to exit. When the AICores in N training cards exit, the AICpus in N training cards exit, ending the communication service.

[0124] It should be noted that the order of the steps of the data synchronization method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of a change within the technical scope disclosed in this application, which should be included in the protection scope of this application, so it will not be repeated.

[0125] The following is an illustration of a computing device in an embodiment of the present application.

[0126] The above describes the data synchronization method of the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a computing device for implementing the data synchronization method. Figures 1 to 6 .based on Figures 1 to 6 The following multiple components are shown: Figures 1 to 6 The computing device shown is capable of executing the above Figure 7 It should be understood that the system may include more additional components than those shown or omit some of the components shown, and the present application embodiment does not limit this. The following description takes the computing device as the first computing device as an example. Figures 1 to 3 As shown, the first computing device 10 may include: a first computing unit 101 and a first communication unit 102 .

[0127] The first computing unit 101 is used to transmit a synchronization preparation request to the first communication unit 102 before obtaining the first data, and the synchronization preparation request instructs the first communication unit 102 to perform a data synchronization operation based on the first data.

[0128] The first communication unit 102 is used to perform a preparation operation required for a data synchronization operation based on a synchronization preparation request.

[0129] The first computing unit 101 is further configured to transmit a synchronization start request to the first communication unit 102 after obtaining the first data, where the synchronization start request instructs the first communication unit 102 to start a data synchronization operation.

[0130] The first communication unit 102 is also used to send a notification to the second communication unit 102 based on the synchronization start request, where the notification indicates that the first communication unit 102 can perform a data synchronization operation, the second communication unit 102 belongs to the second computing device 10, and data synchronization is required between the second computing device 10 and the first computing device 10.

[0131] The first communication unit 102 is further configured to perform a data synchronization operation after the communication units 102 in the plurality of computing devices 10 that need to perform data synchronization with the first computing device 10 are all capable of performing the data synchronization operation.

[0132] Optionally, the first communication unit 102 is further configured to obtain a synchronization start request after completing the preparation operation.

[0133] Alternatively, if Figures 4 to 6As shown, the first computing device 10 further includes: a first message storage area 103. The first computing unit 101 is specifically used to write a synchronization preparation request to the first message storage area 103, and the first communication unit 102 reads the synchronization preparation request from the first message storage area 103. And / or, the first computing unit 101 is specifically used to write a synchronization start request to the first message storage area 103, and the first communication unit 102 reads the synchronization start request from the first message storage area 103.

[0134] Optionally, the first communication unit 102 performs a data synchronization operation, specifically including: operating on the first data so that the second communication unit 102 can obtain the first data; obtaining second data from the second communication unit 102, the second data being the data on which the second communication unit 102 performs the data synchronization operation; and providing first synchronization data to the first computing unit 101, the first synchronization data being obtained by the first communication unit 102 based on the second data.

[0135] Optionally, the first communication unit 102 is further configured to perform a specified operation based on the second data to obtain first synchronization data.

[0136] Optionally, the first communication unit 102 is further configured to perform a specified operation based on the first data and the second data to obtain first synchronization data.

[0137] Optionally, the designated operation is instructed by the first computing unit 101 .

[0138] Optionally, the specified operation includes one or more of the following: a mathematical operation and a rearrangement operation.

[0139] Optionally, the synchronization preparation request carries a destination storage address for storing the first synchronization data. Then the first communication unit 102 is specifically configured to store the first synchronization data in the destination storage address, so that the first calculation unit 101 obtains the first synchronization data from the destination storage address.

[0140] Optionally, the synchronization preparation request carries a source storage address for storing the first data. Then the first communication unit 102 is further configured to obtain the first data from the source storage address.

[0141] Alternatively, if Figures 4 to 6 As shown, the first computing device 10 further includes: a first shared storage area 104, and the first communication unit 102 and the second communication unit 102 can both read data stored in the first shared storage area 104. The first communication unit 102 is specifically used to store the first data in the first shared storage area 104, so that the second communication unit 102 can read the first data from the first shared storage area 104.

[0142] Alternatively, if Figures 4 to 6As shown, the second computing device 10 further includes: a second shared storage area 104, and both the first communication unit 102 and the second communication unit 102 can read data stored in the second shared storage area 104. The first communication unit 102 is specifically used to read the second data from the second shared storage area 104 associated with the second communication unit 102, and the second shared storage area 104 associated with the second communication unit 102 is the shared storage area 104 to which the second communication unit 102 writes the second data.

[0143] Optionally, the first communication unit 102 is further used to obtain a synchronization start request transmitted by the first computing unit 101; the first communication unit 102 is further used to determine whether the second communication unit 102 is capable of performing a data synchronization operation.

[0144] Here, for the detailed working process of the first computing unit 101, the first communication unit 102, the message storage area 103 and the shared storage area 104, and the effects that can be achieved, please refer to the relevant descriptions above. For example, the first computing unit 101 transmits a synchronization preparation request to the first communication unit 102 in the manner described in step 701 above, and transmits a synchronization start request to the first communication unit 102 in the manner described in step 703 above. The first communication unit 102 performs the preparation operations required for the data synchronization operation in the manner described in step 702 above, and performs the data synchronization operation in the manner described in steps 705 and 706 above. The embodiments of the present application will not be repeated here.

[0145] The following is an illustration of the basic hardware structure involved in the embodiments of the present application.

[0146] The embodiment of the present application provides a computing device, which is used to implement part or all of the functions of the data synchronization method provided in the embodiment of the present application. Fig.10 Schematic diagram of a computing device provided in an embodiment of the present application. Fig.10 As shown, the computing device 1000 includes a processor 1001, a memory 1002, a communication interface 1003 and a bus 1004. The processor 1001, the memory 1002 and the communication interface 1003 are connected to each other through the bus 1004.

[0147] Processor 1001 may include a general processor and / or a dedicated hardware chip. A general processor may include: a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). The CPU is, for example, a single-core processor (single-CPU), or a multi-core processor (multi-CPU). A dedicated hardware chip is a hardware module for high-performance processing. A dedicated hardware chip includes at least one of a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or a network processor (NP). Processor 1001 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some or all of the functions of the data synchronization method of the present application may be completed by an integrated logic circuit of hardware in processor 1001 or instructions in software form. By way of example, the above-mentioned computing unit and communication unit may be implemented by processor 1001, respectively.

[0148] The memory 1002 is used to store computer programs, and the computer programs include an operating system 1002a and executable codes (i.e., program instructions) 1002b. The memory 1002 is, for example, a read-only memory or other types of static storage devices that can store static information and instructions, or a random access memory or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a blue-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired executable code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory 1002 is used to store an outbound port queue, etc. The memory 1002 is, for example, independent and connected to the processor 1001 via a bus 1004. Or the memory 1002 and the processor 1001 are integrated together. The memory 1002 can store executable code. When the executable code stored in the memory 1002 is executed by the processor 1001, the processor 1001 is used to perform some or all of the functions of the data synchronization method provided in the embodiment of the present application. For the implementation of the process executed by the processor 1001, please refer to the relevant description in the aforementioned embodiment. The memory 1002 may also include software modules and data required for other running processes such as an operating system. By way of example, the above-mentioned message storage area, shared storage area, and data storage area may be implemented by the memory 1002, respectively.

[0149] The communication interface 1003 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks. For example, the communication interface 1003 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0150] The bus 1004 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., the memory 1002, the processor 1001, and the communication interface 1003). For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the computing device through the bus 1004 as an example. Optionally, the above-mentioned devices inside the computing device 1000 can also be connected to each other in communication with each other using other connection methods other than the bus 1004. For example, the above-mentioned devices inside the computing device 1000 are interconnected through an internal logical interface.

[0151] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0152] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing a program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, all or part of the functions of the data synchronization method provided in the embodiments of the present application are implemented.

[0153] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0154] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0155] Optionally, the structure of at least one computing device included in the computing device cluster can be found in Fig.10 The computing device 1000 is shown. The memory 1002 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the data synchronization method.

[0156] In some possible implementations, the memory 1002 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the data synchronization method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the data synchronization method.

[0157] It should be noted that the memory 1002 in different computing devices 1000 in the computing device cluster can store different instructions, which are respectively used to execute the functions of different computing devices in the data synchronization method provided in the embodiment of the present application. That is, the instructions stored in the memory 1002 in different computing devices 1000 can implement the functions of the first computing device and the second computing device.

[0158] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11 As shown, two computing devices 1100A and 1100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, computing devices 1100A and 1100B include a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The memory 1106 in the computing device 1100A stores instructions for executing the functions of the first computing unit and the first communication unit. At the same time, the memory 1106 in the computing device 1100B stores instructions for executing the functions of the second computing unit and the second communication unit.

[0159] It should be understood that Fig.11 The functions of the computing device 1100A shown in the figure can also be completed by multiple computing devices 1100. Similarly, the functions of the computing device 1100B can also be completed by multiple computing devices 1100. And the deployment mode of the modules used to implement the data synchronization method in the computing device can also be adjusted according to the application requirements.

[0160] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements a data synchronization method as provided in an embodiment of the present application.

[0161] The embodiment of the present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer implements the data synchronization method provided by the embodiment of the present application.

[0162] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0163] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the original data and executable code involved in this application are all obtained with full authorization.

[0164] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means two or more, unless otherwise expressly defined.

[0165] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0166] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data synchronization method, characterized in that: The method is applied to multiple computing devices, and the method includes: Before obtaining the first data, the first computing unit transmits a synchronization preparation request to the first communication unit, wherein the synchronization preparation request instructs the first communication unit to perform a data synchronization operation based on the first data, the first computing device includes the first computing unit and the first communication unit, and the first computing device is any one of the multiple computing devices; The first communication unit performs a preparation operation required for the data synchronization operation based on the synchronization preparation request; After obtaining the first data, the first computing unit transmits a synchronization start request to the first communication unit, wherein the synchronization start request instructs the first communication unit to start the data synchronization operation; The first communication unit sends a notification to the second communication unit based on the synchronization start request, wherein the notification indicates that the first communication unit can perform the data synchronization operation, the second computing device includes a second computing unit and the second communication unit, and the second computing device is any one of the multiple computing devices except the first computing device; The first communication unit performs the data synchronization operation after the communication units in the plurality of computing devices are all capable of performing the data synchronization operation.

2. The method according to claim 1, characterized in that Before the first communication unit sends a notification to the second communication unit based on the synchronization start request, the method further includes: After completing the preparation operation, the first communication unit obtains the synchronization start request.

3. The method according to claim 1 or 2, characterized in that The first computing device also includes: a first message storage area; The first computing unit transmits a synchronization preparation request to the first communication unit, comprising: The first computing unit writes the synchronization preparation request into the first message storage area, and the first communication unit reads the synchronization preparation request from the first message storage area; And / or, the first computing unit transmitting a synchronization start request to the first communication unit includes: The first computing unit writes the synchronization start request to the first message storage area, and the first communication unit reads the synchronization start request from the first message storage area.

4. The method according to any one of claims 1 to 3, characterized in that: The first communication unit performs the data synchronization operation, including: The first communication unit operates the first data so that the second communication unit can obtain the first data; The first communication unit acquires second data from the second communication unit, where the second data is data based on which the second communication unit performs the data synchronization operation; The first communication unit provides first synchronization data to the first calculation unit, where the first synchronization data is obtained by the first communication unit based on the second data.

5. The method according to claim 4, characterized in that The method further comprises: The first communication unit performs a designated operation based on the second data to obtain the first synchronization data.

6. The method according to claim 4, characterized in that The method further comprises: The first communication unit performs a designated operation based on the first data and the second data to obtain the first synchronization data.

7. The method according to claim 5 or 6, characterized in that The designated operation is instructed by the first computing unit.

8. The method according to claim 7, characterized in that The specified operation includes one or more of the following: mathematical operation and rearrangement operation.

9. The method according to any one of claims 4 to 8, characterized in that: The synchronization preparation request carries a destination storage address for storing the first synchronization data, and the first communication unit provides the first synchronization data to the first computing unit, including: The first communication unit stores the first synchronization data in the destination storage address, so that the first calculation unit obtains the first synchronization data from the destination storage address.

10. The method according to any one of claims 4 to 9, characterized in that: The synchronization preparation request carries a source storage address for storing the first data. Before the first communication unit operates on the first data, the method further includes: The first communication unit obtains the first data from the source storage address.

11. The method according to any one of claims 4 to 10, characterized in that: The first computing device further includes: a first shared storage area, the first communication unit and the second communication unit are both capable of reading data stored in the first shared storage area, and the first communication unit operates on the first data so that the second communication unit can obtain the first data, including: The first communication unit stores the first data in the first shared memory area, so that the second communication unit can read the first data from the first shared memory area.

12. The method according to any one of claims 4 to 11, characterized in that: The second computing device further includes: a second shared storage area, the first communication unit and the second communication unit are both capable of reading data stored in the second shared storage area, and the first communication unit obtains second data from the second communication unit respectively, including: The first communication unit reads the second data from a second shared storage area associated with the second communication unit, where the second shared storage area associated with the second communication unit is the shared storage area into which the second communication unit writes the second data.

13. The method according to any one of claims 1 to 12, characterized in that: Before the first communication unit performs the data synchronization operation, the method further includes: The first communication unit obtains a synchronization start request transmitted by the first computing unit; The first communication unit determines that the second communication unit is capable of performing the data synchronization operation.

14. A first computing device, characterized in that: The first computing device includes: a first computing unit and a first communication unit; The first computing unit is used to transmit a synchronization preparation request to the first communication unit before obtaining the first data, wherein the synchronization preparation request instructs the first communication unit to perform a data synchronization operation based on the first data; The first communication unit is used to perform a preparation operation required for the data synchronization operation based on the synchronization preparation request; The first computing unit is further configured to transmit a synchronization start request to the first communication unit after obtaining the first data, wherein the synchronization start request instructs the first communication unit to start the data synchronization operation; The first communication unit is further configured to send a notification to the second communication unit based on the synchronization start request, wherein the notification indicates that the first communication unit can perform the data synchronization operation, the second communication unit belongs to a second computing device, and data synchronization needs to be performed between the second computing device and the first computing device; The first communication unit is further configured to perform the data synchronization operation after the communication units in a plurality of computing devices that need to perform data synchronization with the first computing device are all capable of performing the data synchronization operation.

15. The first computing device of claim 14, wherein: The first communication unit is further configured to obtain the synchronization start request after completing the preparation operation.

16. The first computing device according to claim 14 or 15, characterized in that The first computing device also includes: a first message storage area; The first computing unit is specifically configured to write the synchronization preparation request into the first message storage area, and the first communication unit reads the synchronization preparation request from the first message storage area; And / or, the first computing unit is specifically used to write the synchronization start request into the first message storage area, and the first communication unit reads the synchronization start request from the first message storage area.

17. The first computing device according to any one of claims 14 to 16, characterized in that: The first communication unit is specifically configured to operate the first data so that the second communication unit can obtain the first data; The first communication unit is specifically configured to obtain second data from the second communication unit, where the second data is data based on which the second communication unit performs the data synchronization operation; The first communication unit is specifically configured to provide first synchronization data to the first calculation unit, where the first synchronization data is obtained by the first communication unit based on the second data.

18. The first computing device of claim 17, wherein: The first communication unit is further configured to perform a specified operation based on the second data to obtain the first synchronization data.

19. The first computing device of claim 17, wherein: The first communication unit is further configured to perform a specified operation based on the first data and the second data to obtain the first synchronization data.

20. The first computing device according to claim 18 or 19, characterized in that The designated operation is instructed by the first computing unit.

21. The first computing device of claim 20, wherein: The specified operation includes one or more of the following: mathematical operation and rearrangement operation.

22. The first computing device according to any one of claims 17 to 21, characterized in that: The synchronization preparation request carries a destination storage address for storing the first synchronization data; The first communication unit is specifically configured to store the first synchronization data in the destination storage address, so that the first calculation unit obtains the first synchronization data from the destination storage address.

23. The first computing device according to any one of claims 17 to 22, characterized in that: The synchronization preparation request carries a source storage address for storing the first data; The first communication unit is further configured to obtain the first data from the source storage address.

24. The first computing device according to any one of claims 17 to 23, characterized in that: The first computing device further includes: a first shared storage area, and the first communication unit and the second communication unit are both capable of reading data stored in the first shared storage area; The first communication unit is specifically configured to store the first data in the first shared memory area, so that the second communication unit can read the first data from the first shared memory area.

25. The first computing device according to any one of claims 17 to 24, characterized in that: The second computing device further includes: a second shared storage area, and the first communication unit and the second communication unit can both read data stored in the second shared storage area; The first communication unit is specifically used to read the second data from a second shared storage area associated with the second communication unit, and the second shared storage area associated with the second communication unit is a shared storage area into which the second communication unit writes the second data.

26. The first computing device according to any one of claims 14 to 25, characterized in that: The first communication unit is further used to obtain a synchronization start request transmitted by the first computing unit; The first communication unit is further configured to determine that the second communication unit is capable of performing the data synchronization operation.

27. A computing device, characterized in that The method comprises a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions so that the computing device executes any one of the methods of claims 1 to 13.

28. A computing device cluster, characterized in that: The method comprises a plurality of computing devices, wherein the plurality of computing devices comprises a plurality of processors and a plurality of memories, wherein program instructions are stored in the plurality of memories, and the plurality of processors execute the program instructions, so that the computing device cluster executes any one of the methods described in claims 1 to 13.

29. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 13.

30. A computer-readable storage medium, characterized in that: The method comprises program instructions, and when the program instructions are executed on a computing device, the computing device is caused to execute the method according to any one of claims 1 to 13.

Images