Data migration method and device

Abstract

The data migration method comprises the steps that a data subscription request sent by at least one first process is received, one data subscription request carries a data identifier of at least one piece of target data, one first process runs on one first accelerator card, and one piece of target data is data to be read by one first process; the data identifiers of the target data are paired with data identifiers of source data recorded in the server side, one piece of source data is stored on one second acceleration card and generated by one second process, and one second process runs on one second acceleration card; and when the data identifier of the at least one piece of target data carried in the first data subscription request is successfully paired with the data identifier of the source data recorded in the server side, issuing an event notification to the processes related to the successfully paired identifier, so as to transmit data between acceleration cards where the processes are located. The method breaks through the limitation of underlying hardware and a set communication software library, and the data communication efficiency is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of information technology (IT), and in particular to a data migration method and device. Background Art

[0002] In multi-process, multi-threaded, distributed systems, or resource-constrained environments, data production and consumption dependencies often form dynamically between application processes due to factors such as task complexity, resource optimization requirements, and the real-time and dynamic nature of data. For example, in high-performance computing, complex scientific computing tasks (such as weather simulations and molecular dynamics simulations) require multiple processes to work together. Some processes act as data producers, generating intermediate data, while other processes act as data consumers, requiring this data for further calculations. Similarly, in parallel matrix computations, some processes compute partial matrix results, while other processes rely on these results for subsequent calculations. In large language model (LLM) inference scenarios, some processes generate data during the prefill phase, while other processes require this data during the decode phase.

[0003] When the data producer and consumer processes run on different accelerator cards (such as neural processing units (NPUs), tensor processing units (TPUs), graphics processing units (GPUs), etc.), the data generated by the producer process needs to be transferred to the AI ​​accelerator memory where the consumer process is located so that the consumer process can use the data for further calculations. This dynamic production and consumption dependency requires efficient and flexible data communication between processes. In order to improve the efficiency of data communication in such scenarios, the industry generally adopts high-performance collective communication libraries to accelerate communication between processes. However, this approach is usually subject to the constraints of the underlying hardware and collective communication software libraries, resulting in low data communication efficiency and difficulty in adapting to dynamic communication needs. For example, the network bandwidth and latency of the underlying hardware limit the data transmission speed, and the fixed mode of the communication library cannot flexibly cope with dynamic changes in complex applications. Summary of the Invention

[0004] The present application provides a data migration method, device, computer storage medium and computer product, which can reduce the task processing delay in server-unaware computing.

[0005] In a first aspect, the present application provides a data migration method, which is applied to a server, comprising: receiving a first data subscription request sent from at least one first process, wherein a first data subscription request carries a data identifier of at least one target data, a first process runs on a first accelerator card, a target data is data to be read by a first process, and the target data related to the same first data subscription request is read by the same first process; pairing the data identifier of each target data with the data identifier of the source data recorded in the server, wherein a source data is stored on a second accelerator card and generated by a second process, and a second process runs on the second accelerator card; the data identifier carried in the second data subscription request When the data identifier of at least one target data is successfully paired with the data identifier of the source data recorded in the server, an event notification is sent to the third process and at least one fourth process; wherein, the second data subscription request is any first data subscription request received by the server, the third process is the process that sends the second data subscription request in at least one first process, the fourth process is the process that generates the first source data, and the data identifier of the first source data is the data identifier of the source data that is successfully paired with the data identifier of the target data in the second data subscription request; the event notification is used to notify the third process and the fourth process to send a communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process. In this way, through the P2P communication mechanism based on data publishing and subscription, the limitations of the underlying hardware and collective communication software library are broken through, the data communication efficiency is significantly improved, and at the same time, it is flexibly adapted to dynamic communication needs.

[0006] In one possible implementation, sending an event notification to a third process includes sending a first event notification to the third process, wherein the first event notification is used to notify the third process to send at least one data receiving operator to a first accelerator card running the third process; wherein each data receiving operator is used to receive one piece of first source data, and if there are multiple pieces of first source data, the order in which the data receiving operators are sent is the same as the order in which the first source data are paired. In this way, the order of data transmission can be ensured, and P2P communication timing control can be implemented.

[0007] In one possible implementation, sending an event notification to at least one fourth process includes: when there are multiple paired first source data and they come from different fourth processes, sending a second event notification to each fourth process in sequence according to the order of the paired first source data, wherein one second event notification is used to notify one fourth process to send a first data sending operator to the first accelerator card running the fourth process, wherein one first data sending operator is used to send one first source data; when there are multiple paired first source data and they come from one fourth process, sending a third event notification to the fourth process, wherein the third event notification is used to notify the fourth process to send multiple second data sending operators to the first accelerator card running the fourth process, wherein one second data sending operator is used to send one first source data, and the sending order of the second data sending operators is the same as the order of the paired first source data. In this way, the order of data transmission can be ensured and P2P communication timing control can be achieved.

[0008] In one possible implementation, the method further includes: receiving first root information generated by the third process when a communication domain has not been established between the runtime environment of the third process and the runtime environment of the fourth process, and sending the first root information to the fourth process, wherein the first root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process; or receiving second root information generated by the fourth process when a communication domain has not been established between the runtime environment of the third process and the runtime environment of the fourth process, and sending the second root information to the third process, wherein the second root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process. In this way, dynamic creation of communication domains can be achieved.

[0009] In one possible implementation, if the data identifier carried in the third data subscription request is not successfully paired, the server continues to wait for pairing with the data identifier carried in the third data subscription request within a set pairing timeout period until pairing is successful or the pairing timeout period is reached. The third data subscription request is any first data subscription request received by the server. This avoids frequent transmission of data subscription requests and conserves communication resources.

[0010] In one possible implementation, before sending event notifications to the third process and the fourth process respectively, the method further includes: determining that the size of a data block storing the first target data on the first memory is the same as the size of a data block storing source data related to the first target data on the second memory, and that the number of data blocks storing the first target data on the first memory is the same as the number of data blocks storing source data related to the first target data on the second memory, wherein the first memory is the memory of a first accelerator card running the third process, the second memory is the memory of a second accelerator card running the fourth process, and the first target data is any target data subscribed to by the second data subscription request. This ensures that data can be transmitted successfully.

[0011] In one possible implementation, after sending the event notification to the third process and the fourth process respectively, the process further includes: clearing the source data information related to the successfully paired data identifier recorded by the server. In this way, the source data information can be dynamically managed.

[0012] In the second aspect, the present application provides a data migration method, which is applied to a first accelerator card, and a first process is running on the first accelerator card. The method includes: calling a data subscription interface through the first process to send a data subscription request to the server, wherein the data subscription request carries a data identifier of at least one target data, and the target data is the data to be read by the first process; receiving an event notification from the server, wherein the event notification is sent when the data identifier of the target data and the data identifier of at least one source data recorded by the server are successfully paired, and the data identifier of one source data is sent to the server by a second process, and the second process runs on a second accelerator card; in response to the event notification, calling an operator sending interface through the first process to send at least one data receiving operator to the collective communication library on the first accelerator card, wherein the data receiving operator is used to receive the first source data from the memory side of the second accelerator card storing the first source data, and store it in the memory of the first accelerator card, and the first source data is the source data related to the data identifier that is successfully paired. In this way, when a process needs to consume target data, it can send a data subscription request to the server in real time or periodically to obtain the target data from other accelerator cards. Therefore, through the P2P communication mechanism based on data publishing and subscription, the limitations of the underlying hardware and collective communication software library are broken through, the data communication efficiency is significantly improved, and it can flexibly adapt to dynamic communication needs.

[0013] In a possible implementation, the order in which the multiple data receiving operators are generated is the same as the order in which the data identifiers are successfully paired in the data subscription request.

[0014] In one possible implementation, the operator sending interface is called by the first process to send a data receiving operator to the collective communication library on the first accelerator card, including: in the presence of a reusable first communication domain, using the thread of the first communication domain to be created in the thread pool in the first process, calling the operator sending interface, and sending the data receiving operator to the collective communication library on the first accelerator card, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the third process, and the third process is the process that generates the first source data.

[0015] In one possible implementation, the operator sending interface is called by the first process to send the data receiving operator to the collective communication library on the first accelerator card, including: in the absence of a reusable first communication domain, using the idle thread in the thread pool in the first process to create the first communication domain, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the third process, and the third process is the process that generates the first source data; using the thread that creates the first communication domain in the thread pool, calling the operator sending interface, and sending the data receiving operator to the collective communication library on the first accelerator card.

[0016] In one possible implementation, an idle thread in a thread pool in a first process is used to create a first communication domain, including: using the idle thread to send a root information query request to a server, wherein the root information query request is used to query root information generated by a third process, wherein the third process is a process that generates first source data; using the idle thread to receive root information returned from the server; and using the idle thread to create the first communication domain based on the root information.

[0017] In a possible implementation, the method further includes: caching the first communication domain, recording an identifier of a runtime environment of the third process, and recording a thread that creates the first communication domain.

[0018] In one possible implementation, after creating the first communication domain using an idle thread in a thread pool in the first process, the method further includes: performing at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link on the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

[0019] In a possible implementation, the method further includes: in a process of using the first communication domain to transmit the first source data, using idle threads in a thread pool in parallel to create a communication domain between the runtime environment of the first process and other runtime environments.

[0020] In a possible implementation, the number of threads in the thread pool is the number of concurrent communication domains supported by the first accelerator card.

[0021] In a third aspect, the present application provides a data migration method, which is applied to a second accelerator card, on which a second process is running, and the data generated by the second process includes at least one first source data, and the first source data is stored in the memory of the second accelerator card. The method includes: receiving an event notification from the server, wherein the event notification is sent when the data identifier of the first target data and the data identifier of the first source data recorded by the server are successfully paired, and the data identifier of the first target data is sent to the server by the first process; in response to the event notification, calling the operator sending interface through the second process, and sending at least one data sending operator to the collective communication library on the second accelerator card, wherein the data sending operator is used to send the first source data to the memory of the first accelerator card running the first process.

[0022] In one possible implementation, one data sending operator is used to send one first source data, and multiple data sending operators send data to the same first accelerator card, wherein the generation order of the data sending operators is the same as the order in which the first source data are paired.

[0023] In one possible implementation, an operator is called through a second process to send a data sending operator to a collective communication library on a second accelerator card, including: in the presence of a reusable first communication domain, a thread of the first communication domain is created in a thread pool in the second process, an operator sending interface is called, and a data sending operator is sent to the collective communication library on the second accelerator card, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the second process.

[0024] In one possible implementation, the operator sending interface is called by the second process to send the data sending operator to the collective communication library on the second accelerator card, including: in the absence of a reusable first communication domain, using the idle thread in the thread pool in the second process to create the first communication domain, wherein the first communication domain is the communication domain established between the runtime environment of the first process and the runtime environment of the second process; using the thread that creates the first communication domain in the thread pool to call the operator sending interface and send the data sending operator to the collective communication library on the second accelerator card.

[0025] In one possible implementation, an idle thread in a thread pool in a second process is used to create a first communication domain, including: using the idle thread to send a root information query request to a server, wherein the root information query request is used to query the root information generated by the first process; using the idle thread to receive the root information returned from the server; and using the idle thread to create the first communication domain based on the root information.

[0026] In a possible implementation, after creating the first communication domain using an idle thread in the thread pool, the method further includes: performing at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link on the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

[0027] In a possible implementation, the method further includes: in the process of transmitting the first source data using the first communication domain, using idle threads in a thread pool in parallel to create a communication domain between the runtime environment of the second process and other runtime environments.

[0028] In a possible implementation, the method further includes: caching the first communication domain, recording an identifier of the runtime environment of the first process, and recording a thread that creates the first communication domain.

[0029] In one possible implementation, the number of threads in the thread pool is the number of concurrent communication domains supported by the second AI accelerator card.

[0030] In a fourth aspect, the present application provides a data migration device, which can be but is not limited to being deployed on the aforementioned server. The data migration device includes: a communication module and a processing module. The communication module is used to receive a first data subscription request sent from at least one first process, wherein a first data subscription request carries a data identifier of at least one target data, a first process runs on a first accelerator card, a target data is data to be read by a first process, and the target data related to the same first data subscription request is read by the same first process. The processing module is used to pair the data identifiers of each target data with the data identifiers of the source data recorded in the server, wherein a source data is stored on a second accelerator card and generated by a second process, and a second process runs on a second accelerator card. The communication module is also used to send an event notification to the third process and at least one fourth process when the data identifier of at least one target data carried in the second data subscription request is successfully paired with the data identifier of the source data recorded in the server. Among them, the second data subscription request is any first data subscription request received by the server, the third process is the process that sends the first data subscription request in at least one first process, the fourth process is the process that generates the first source data, and the data identifier of the first source data is the data identifier of the source data that is successfully paired with the data identifier of the target data in the second data subscription request; the event notification is used to notify the third process and the fourth process to send the communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process.

[0031] In some possible implementations, the processing module sends an event notification to the third process, including: sending a first event notification to the third process, wherein the first event notification is used to notify the third process to send at least one data receiving operator to the first accelerator card running the third process; wherein one data receiving operator is used to receive one first source data, and when there are multiple first source data, the order in which the data receiving operators are sent is the same as the order in which the first source data are paired.

[0032] In some possible implementations, the processing module sends an event notification to at least one fourth process, including: when there are multiple paired first source data and they come from different fourth processes, sending a second event notification to each fourth process in sequence according to the order of the paired first source data, wherein one second event notification is used to notify one fourth process to send a first data sending operator to the first accelerator card running the fourth process, wherein one first data sending operator is used to send one first source data; when there are multiple paired first source data and they come from one fourth process, sending a third event notification to the fourth process, wherein the third event notification is used to notify the fourth process to send multiple second data sending operators to the first accelerator card running the fourth process, wherein one second data sending operator is used to send one first source data, and the sending order of the second data sending operators is the same as the order of the paired first source data.

[0033] In some possible implementations, the communication module is further used to: when a communication domain is not established between the runtime environment of the third process and the runtime environment of the fourth process, receive first root information generated from the third process, and send the first root information to the fourth process, wherein the first root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process; or, when a communication domain is not established between the runtime environment of the third process and the runtime environment of the fourth process, receive second root information generated from the fourth process, and send the second root information to the third process, wherein the second root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process.

[0034] In some possible implementations, the processing module is further configured to: if the data identifier carried by the third data subscription request is not successfully paired, continue to wait for pairing of the data identifier carried by the third data subscription request within a set pairing timeout period until the pairing is successful or the pairing timeout period is reached, wherein the third data subscription request is any one of the first data subscription requests received by the server.

[0035] In some possible implementations, before the processing module sends the event notification to the third process and the fourth process respectively, it is further used to: determine that the size of the data block storing the first target data on the first memory is the same as the size of the data block storing the source data related to the first target data on the second memory, and the number of data blocks storing the first target data on the first memory is the same as the number of data blocks storing the source data related to the first target data on the second memory, wherein the first memory is the memory of the first accelerator card running the third process, the second memory is the memory of the second accelerator card running the fourth process, and the first target data is any target data subscribed to by the second data subscription request.

[0036] In some possible implementations, after sending the event notification to the third process and the fourth process respectively, the processing module is further configured to: clear information of source data related to the data identifier of successful pairing recorded by the server.

[0037] In a fifth aspect, the present application provides a data migration device, which can be, but is not limited to, deployed on the aforementioned first accelerator card, on which a first process is running. The data migration device includes: a communication module and a processing module. The communication module is configured to call a data subscription interface through the first process to send a data subscription request to a server, wherein the data subscription request carries a data identifier of at least one target data, the target data being the data to be read by the first process. The communication module is further configured to receive an event notification from the server, wherein the event notification is sent when the data identifier of the target data is successfully matched with the data identifier of at least one source data recorded by the server, the data identifier of the source data being sent to the server by a second process, the second process running on a second accelerator card. The processing module is configured to, in response to the event notification, call an operator delivery interface through the first process to deliver at least one data receiving operator to the collective communication library on the first accelerator card, wherein the data receiving operator is configured to receive first source data from the memory of the second accelerator card storing the first source data and store it in the memory of the first accelerator card, wherein the first source data is source data associated with the successfully matched data identifier.

[0038] In some possible implementations, the generation order of multiple data receiving operators is the same as the order of successfully paired data identifiers in the data subscription request.

[0039] In some possible implementations, when the processing module calls the operator delivery interface through the first process and delivers the data receiving operator to the collective communication library on the first accelerator card, it is specifically used to: in the presence of a reusable first communication domain, use the thread of the first communication domain created in the thread pool in the first process, call the operator delivery interface, and deliver the data receiving operator to the collective communication library on the first accelerator card, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the third process, and the third process is the process that generates the first source data.

[0040] In some possible implementations, when the processing module calls the operator delivery interface through the first process and delivers the data receiving operator to the collective communication library on the first accelerator card, it is specifically used to: in the absence of a reusable first communication domain, use the idle thread in the thread pool in the first process to create the first communication domain, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the third process, and the third process is the process that generates the first source data; use the thread that creates the first communication domain in the thread pool to call the operator delivery interface and deliver the data receiving operator to the collective communication library on the first accelerator card.

[0041] In some possible implementations, when the processing module uses an idle thread in a thread pool in a first process to create a first communication domain, it is specifically used to: use the idle thread to send a root information query request to the server, wherein the root information query request is used to query the root information generated by a third process, wherein the third process is the process that generates the first source data; use the idle thread to receive the root information returned from the server; and use the idle thread to create the first communication domain based on the root information.

[0042] In some possible implementations, the processing module is further configured to: cache the first communication domain, record an identifier of a runtime environment of the third process, and record a thread that creates the first communication domain.

[0043] In some possible implementations, after the processing module uses an idle thread in a thread pool in the first process to create the first communication domain, it is further used to: perform at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link on the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

[0044] In some possible implementations, the processing module is further configured to: in a process of using the first communication domain to transmit the first source data, in parallel use idle threads in the thread pool to create a communication domain between the runtime environment of the first process and other runtime environments.

[0045] In some possible implementations, the number of threads in the thread pool is the number of concurrent communication domains supported by the first accelerator card.

[0046] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0047] In a sixth aspect, the present application provides a data migration device, which can be but is not limited to being deployed on the aforementioned second accelerator card, on which a second process is running, and the data generated by the second process includes at least one first source data, and the first source data is stored in the memory of the second accelerator card. The data migration device includes: a communication module and a processing module. The communication module is used to receive an event notification from the server, wherein the event notification is sent when the data identifier of the first target data and the data identifier of the first source data recorded by the server are successfully paired, and the data identifier of the first target data is sent to the server by the first process. The processing module is used to respond to the event notification, call the operator sending interface through the second process, and send at least one data sending operator to the collective communication library on the second accelerator card, wherein the data sending operator is used to send the first source data to the memory of the first accelerator card running the first process.

[0048] In some possible implementations, one data sending operator is used to send one first source data, and multiple data sending operators send data to the same first accelerator card, wherein the generation order of the data sending operators is the same as the order in which the first source data are paired.

[0049] In some possible implementations, when the processing module calls the operator through the second process and sends the data sending operator to the collective communication library on the second accelerator card, it is specifically used to: in the presence of a reusable first communication domain, use the thread of the first communication domain created in the thread pool in the second process, call the operator sending interface, and send the data sending operator to the collective communication library on the second accelerator card, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the second process.

[0050] In some possible implementations, when the processing module calls the operator through the second process and sends the data sending operator to the collective communication library on the second accelerator card, it is specifically used to: in the absence of a reusable first communication domain, use the idle thread in the thread pool in the second process to create a first communication domain, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the second process; use the thread that creates the first communication domain in the thread pool to call the operator sending interface to send the data sending operator to the collective communication library on the second accelerator card.

[0051] In some possible implementations, when the processing module uses an idle thread in a thread pool in a second process to create a first communication domain, it is specifically used to: use the idle thread to send a root information query request to the server, wherein the root information query request is used to query the root information generated by the first process; use the idle thread to receive the root information returned from the server; and use the idle thread to create the first communication domain based on the root information.

[0052] In some possible implementations, after the processing module uses an idle thread in the thread pool to create the first communication domain, it is further used to: perform at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link on the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

[0053] In some possible implementations, the processing module is further configured to: in a process of transmitting the first source data using the first communication domain, in parallel use idle threads in the thread pool to create a communication domain between the runtime environment of the second process and other runtime environments.

[0054] In some possible implementations, the processing module is further configured to cache the first communication domain, record an identifier of a runtime environment of the first process, and record a thread that creates the first communication domain.

[0055] In some possible implementations, the number of threads in the thread pool is the number of concurrent communication domains supported by the second AI accelerator card.

[0056] In the seventh aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing node, the computing node executes the method described in the first aspect or any possible implementation of the first aspect, or executes the method described in the second aspect or any possible implementation of the second aspect, or executes the method described in the third aspect or any possible implementation of the third aspect.

[0057] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing node, causes the computing node to execute the method described in the first aspect or any possible implementation of the first aspect, or to execute the method described in the second aspect or any possible implementation of the second aspect, or to execute the method described in the third aspect or any possible implementation of the third aspect.

[0058] It can be understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a schematic diagram of a KV migration process based on a collective communication library to implement LLM pre-filling and decoding separation architecture provided by an embodiment of the present application;

[0060] Figure 2 This is a schematic diagram of an architecture for implementing KV migration based on Mooncake storage host-side cache, provided in an embodiment of the present application;

[0061] Figure 3 This is a schematic diagram of a data migration scenario provided by an embodiment of the present application;

[0062] Figure 4 This is a technical diagram of data migration provided by an embodiment of the present application;

[0063] Figure 5 This is a schematic diagram of a data migration process using two runtimes as an example provided in an embodiment of the present application;

[0064] Figure 6 This is a schematic diagram of the process of data publishing, subscription, event notification, and operator issuance provided by an embodiment of the present application;

[0065] Figure 7 This is another schematic diagram of a data migration process using two runtimes as an example provided by an embodiment of the present application; Figure 8 This is a schematic diagram of a deadlock provided in an embodiment of the present application;

[0066] Figure 9 This is a schematic diagram of the mapping relationship between threads and operators during runtime provided by an embodiment of the present application;

[0067] Figure 10 This is a schematic diagram of the concurrent design of communication domains and links provided in an embodiment of the present application;

[0068] Figure 11 This is a flow chart of a data migration method provided in an embodiment of the present application;

[0069] Figure 12 This is a flowchart of another data migration method provided in an embodiment of the present application;

[0070] Figure 13 This is a flowchart of another data migration method provided in an embodiment of the present application;

[0071] Figure 14 This is a structural diagram of a data migration device provided in an embodiment of the present application;

[0072] Figure 15 is a structural diagram of another data migration device provided in an embodiment of the present application;

[0073] Figure 16 This is a structural diagram of another data migration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0075] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0076] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0077] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0078] First, the relevant technical terms involved in the technical solution provided in this application are introduced.

[0079] (1) Runtime and Process

[0080] The runtime environment (or "runtime") refers to the environment and state of an application (APP) during runtime, including mechanisms such as memory management, thread scheduling, and garbage collection. A process is a specific instance of an application running in the operating system and a concrete manifestation of the runtime environment. The runtime environment provides the infrastructure necessary for a process to run, while a process is a specific running instance of the runtime environment. In an operating system, an instance is a process running during the application's runtime, and each process is a running instance of an application.

[0081] (2) Communication Operator

[0082] Communication operators are specific computing tasks that the host runtime sends to the device (such as a GPU or NPU) for execution. These operators typically exist in the form of kernels, which are program fragments running on the device and are specifically designed to perform specific computing operations. Communication operators are primarily used to transmit data and can be divided into two categories: data sending operators and data receiving operators. Data sending operators are primarily responsible for sending data to data receiving operators, while data receiving operators are primarily responsible for receiving data from data sending operators.

[0083] (3) Communication operator dispatch

[0084] The communication operator is sent to the host side during runtime, which means that the host side sends the communication-related operations (i.e., the communication operator) to the device side for execution, thereby realizing cross-card communication between devices.

[0085] (4) Peer to peer (P2P) communication timing

[0086] P2P communication timing refers to the order in which communication operators dispatch operations from the queue when multiple runtimes communicate. The order in which operators dispatch operations from the sending and receiving operators must be the same. Each sending operator corresponds to a receiving operator, and their order must be strictly consistent. This ensures correct data transmission and avoids data out-of-order and deadlock issues.

[0087] For example, suppose runtime 1 is bound to device A and runtime 2 is bound to device B. Runtime 1 needs to send two data blocks (Data Block 1 and Data Block 2) to runtime 2, and runtime 1 needs to receive one data block (Data Block 3) from runtime 2.

[0088] The expected order of communication operator processing is as follows: the operator queue on the sending end (Device A) is [Send(Data Block 1), Send(Data Block 2), Recv(Data Block 3)]; the operator queue on the receiving end (Device B) is [Recv(Data Block 1), Recv(Data Block 2), Send(Data Block 3)]. To ensure correct communication, the operator order on the sending and receiving ends must be consistent: Runtime 1 first sends the Send(Data Block 1) operator to Device A, then sends the Send(Data Block 2) operator, and finally sends the Recv(Data Block 3) operator; Runtime 2 first sends the Recv(Data Block 1) operator to Device B, then sends the Recv(Data Block 2) operator, and finally sends the Send(Data Block 3) operator. If the timing of operator delivery in Runtime 1 and Runtime 2 is incorrect, two problems may occur:

[0089] 1) Data confusion

[0090] For example, if Device A's sequence is: Send (Data Block 1) -> Send (Data Block 2) -> Recv (Data Block 3); and Device B's sequence is: Recv (Data Block 2) -> Recv (Data Block 1) -> Send (Data Block 3), then Device A's Data Block 1 will be transferred to the storage address of Device B's Data Block 2. However, the receiving end expects to receive data using the storage address of Device B's Data Block 1, resulting in data corruption.

[0091] 2) Deadlock

[0092] For example, if Device A's sequence is Send(Data Block 1)->Send(Data Block 2)->Recv(Data Block 3), and Device B's sequence is Send(Data Block 3)Recv(Data Block 1)->Recv(Data Block 2), then Device A's Send(Data Block 1) operator will wait for Device B to receive data, while Device B's Send(Data Block 3) operator will also wait for Device A to receive data. Consequently, the two operators will wait for each other, resulting in a deadlock.

[0093] (5) Communication domain

[0094] A communication domain is a logical communication environment that defines a group of devices (such as GPUs or NPUs) that can communicate with each other, as well as the communication rules and configurations between them. In distributed computing, communication domains are often used to manage communication between devices, ensuring the correctness and efficiency of communication. In distributed computing, multiple devices (such as GPUs or NPUs) need to communicate with each other to complete tasks. To achieve this communication, the following problems need to be solved: a) How to establish communication connections between devices? b) How to identify and locate communicating devices? c) How to define communication rules and protocols? Communication domains were created to address these issues, providing a unified management and configuration environment for communication between devices.

[0095] (6) Communication domain creation

[0096] Communication domain creation involves establishing a communication domain before enabling P2P communication between two runtimes. This process involves using one runtime as the root and utilizing the addressing information it provides to collaboratively initialize the communication domain between the two runtimes. The process for establishing a communication domain can be as follows: Before enabling P2P communication between multiple runtimes, one runtime must first be selected as the root. The root runtime's role is to provide a unified starting point for initializing the communication domain and addressing information to help other runtimes find each other. The root runtime then provides key addressing information (also known as "root information"), typically including IP addresses and port numbers, device identifiers, and communication protocols. This information is typically communicated to other runtimes using inter-process communication. Other runtimes (non-root runtimes) then connect to the root runtime over the network and exchange necessary information, such as device identifiers and communication protocols. Based on this information, the root runtime establishes a communication connection between the devices and defines communication rules and protocols, such as the data transmission format and error handling mechanisms. Finally, after completing these steps, the communication domain is successfully established. At this point, the communication connection between the devices has been established, the communication rules have been configured, and P2P communication can begin.

[0097] (7) Communication link

[0098] A communication link is the underlying physical or logical connection between two devices (such as a GPU or NPU) for data transmission. It is the foundation of communication and ensures that data can be transmitted efficiently and reliably between devices.

[0099] (8) Communication link establishment

[0100] Communication link creation refers to the creation of the underlying communication link during P2P communication between two runtimes. This process is typically triggered during the first communication within a communication domain and involves steps such as selecting a transport protocol, initializing the link, and testing the link. The creation of a communication link provides the necessary channel for communication between devices, ensuring efficient and reliable data transmission.

[0101] (9) Data Block

[0102] A data block typically refers to a contiguous section of memory allocated in memory for storing data. This data can be variables, arrays, objects, or other data structures in a program. Data blocks in memory can vary in size and type and serve as the foundation for data processing and storage in a program. In this embodiment, a data block refers to a contiguous section of memory allocated in the accelerator card's memory (e.g., high bandwidth memory (HBM)).

[0103] (10) Named Object Publish and Subscribe

[0104] Named object publish and subscribe refers to a runtime interface and data semantics based on the association between named objects and data blocks provided by the data client, which makes it easier for application developers to express the dependencies between data blocks in multiple runtimes.

[0105] Next, the technical solution provided by this application is introduced.

[0106] For example, Figure 1 The figure shows a KV migration process diagram based on the collective communication library to realize LLM pre-filling and decoding separation architecture. Figure 1 As shown in the figure, the process of data communication between application processes is as follows: First, when an inference instance is initialized, the Prefill and Decode roles of the process are determined, and a collective communication domain is established between the Prefiller and Decoder processes. This step defines the number and roles of the Prefill and Decode instances. Then, during inference, an initial hint is sent to the Router, which assigns tasks to the Prefiller based on load. During the Prefiller phase, generated key-value pairs (KVs) are saved and actively transmitted to the Decoder connector via the Prefill connector. After the Prefiller process completes inference, it notifies the Router, which in turn notifies the Decoder process to continue inference processing in the Decode phase. The Decoder receives the KVs sent by the Prefill connector via its connector. Subsequently, further hints are sent to the Router, which assigns tasks to the Decoder. The Decoder loads the KVs from its connector, either in batches or layer by layer, and then performs inference. In this solution, the Router's primary role is task allocation, not order-preserving data transmission. This can lead to timing disruptions, making it unsuitable for scenarios with multiple Prefill and Decode instances. Furthermore, since the number of Prefill and Decode instances is predefined at the initial stage, the system's scalability is limited. Specifically, this solution typically operates in static graph mode, meaning that after system startup, the number and roles of Prefill and Decode instances are fixed and cannot be adjusted based on runtime requirements.

[0107] For example, Figure 2This diagram illustrates an architecture for implementing KV migration based on Mooncake Store host-side caching. The diagram primarily involves the following components: Master Service, vLLM Inference Engine, Mooncake Store Instance, and Transfer Engine. The Master Service, as the primary service, is responsible for control plane management (i.e., it is responsible for control and management tasks) and coordinates task allocation between the Prefil and Decode phases. It is an independent process. The LLM Inference Engine is an inference engine responsible for processing input data and generating output. The Mooncake Store Instance is a host-side cache instance used to store KV pairs. Each Mooncake Store Instance is allocated a certain amount of storage space as part of the Logical Memory Pool. Data transfer is actually from one store to another, without passing through the Master. The diagram shows multiple Mooncake Store instances, each associated with a different vLLM Inference Engine instance. The Transfer Engine is responsible for data transfer between different phases, particularly the migration of KV pairs. The figure shows the Transfer Engine's zero-copy transfer functionality. In this architecture, data generated during the Prefill phase must be copied from the memory on the accelerator card where the Prefill process runs to the Mooncake Store Instance in the Prefill phase. Then, during the Decode phase, the data in the Mooncake Store Instance in the Prefill phase is first copied to the Mooncake Store Instance in the Decode phase, and then copied from the Mooncake Store Instance in the Decode phase to the memory on the accelerator card where the Decode process runs. Therefore, each transfer must pass through the Mooncake Store Instance, performing two copies between the Mooncake Store Instance and the accelerator card's memory space. As the data increases, the copy latency can increase by tens to hundreds of milliseconds, which cannot meet the low-latency data migration requirements of online inference scenarios.In addition, in this architecture, the transmission bandwidth across nodes depends on the host-side network card. The bandwidth is lower than the link between NPU / GPU cards and is easily interfered with by other services, which may further affect the efficiency of data transmission.

[0108] In view of this, an embodiment of the present application provides a data migration method that breaks through the limitations of the underlying hardware and collective communication software library, significantly improves data communication efficiency, and flexibly adapts to dynamic communication needs.

[0109] For example, Figure 3 A schematic diagram of a data migration scenario provided by an embodiment of the present application is shown. Figure 3 As shown, this scenario includes at least one computing node. Different computing nodes can communicate with each other through, but are not limited to, wired or wireless networks, and can work together to complete tasks or provide services. The hardware layer on the computing node can include at least one accelerator card, such as a GPU, NPU, etc. The software layer on the computing node can include at least one runtime and a server (Worker) deployed separately from the runtime. A client (Client) is deployed in each runtime. A runtime is bound to an accelerator card and interacts with the accelerator card through the interface of the collective communication library within the accelerator card. Exemplarily, the binding of the runtime to the accelerator card can be understood as, but is not limited to, the runtime being configured to work in conjunction with the accelerator card hardware to more efficiently execute related computing tasks. For example, the runtime is responsible for issuing operators, while the accelerator card is responsible for executing operators. The collective communication library is primarily used to provide efficient data exchange capabilities for large-scale parallel computing tasks, supporting multiple collective communication primitives and algorithms to optimize collective communication efficiency and improve system performance, thereby accelerating the execution of high-performance computing applications. Exemplarily, the collective communication library can be, but is not limited to, the Huawei collective communication library (HCCL).

[0110] Among them, Client refers to the dynamic library linked in the runtime and the programming interface provided. The type of interface provided by Data Clien can be, but is not limited to, C++, Python interface, etc. The interface provided by Client can include: data publishing interface and data subscription interface. The data publishing interface is mainly used to send the data identifier of the source data to the Worker, and the data subscription interface is mainly used to send the data identifier of the target data to the Worker. Among them, the source data is generated by a process and stored on the accelerator card where the process is located. The target data is consumed by another process, and storage space for the target data is reserved in the memory of the accelerator card where the process is located. The data transmission in this embodiment can be, but is not limited to, direct data transmission between accelerator cards.

[0111] The interface provided by the Client can support different runtimes to trigger the publication and subscription of named objects through the interface, and the address information of the data blocks (blobs) to be transmitted. In addition, the interface provided by the Client can support the return of a Future after the interface is called, so as to obtain the success or failure of each data transmission through the Future. Among them, the data publishing interface and data subscription interface provided by the Client are asynchronous interfaces, that is, a Future object is returned immediately after the interface is called, without waiting for the data transmission to be completed, so that the program can continue to execute subsequent code and query the success or failure of each data transmission through the Future object when needed. Exemplary, the data publishing interface can be as follows:

[0112]

[0113] The data subscription interface can be as follows:

[0114]

[0115] In the aforementioned interfaces, D2D stands for device-to-device, indicating direct data transmission between devices. This implies that the implementation utilizes collective communication and the underlying high-speed link. Named objects establish an association between keys (data identifiers of the data to be transmitted) and the data blocks (blobs) to be sent and received. Futures support asynchronous access to the results of completed data transmissions, facilitating the sender's decision on when to release data and the receiver's decision on when to trigger data consumption. The data identifiers of the data to be transmitted can be, but are not limited to, named by the developer at both the sender and receiver according to established rules to generate corresponding data object names.

[0116] Worker refers to a process that is deployed separately from the runtime and is used to coordinate the dynamic creation of collective communication domains and the timing of concurrent P2P communications. The operation of Worker can be managed by an independent process. Worker contains startup and configuration scripts and binaries to provide naming objects and communication order preservation, and dynamic communication domain creation services. Worker and Client runtime libraries will collaborate to implement the publishing and subscription of data objects, the distribution control of communication operators, and the creation control of dynamic communication domains and communication links. In this embodiment, Workers on different computing nodes can be interconnected in pairs to form a distributed cluster. In order to achieve the expansion and reduction of Worker services, Workers can be managed through the ETCD service. Figure 3In the distributed scenario shown, all data subscription requests and data publishing requests will first be sent to the Worker in the computing node where the runtime is located. The Worker will then forward the request to the Worker responsible for object metadata management (i.e., the Master). The Worker responsible for object metadata management is selected in the entire cluster by performing hash calculations on the same value. That is, the entire cluster will select a Worker to assume the role of the Master for metadata management. For ease of description, the following will no longer distinguish between non-Masters and Masters. At this time, the relationship between Client and Worker can be as follows: Figure 4 shown.

[0117] Continue reading Figure 4 During the execution of a process at runtime, when the process is a data producer, the process can call the data publishing interface in the Client configured therein to send a named object publishing request (or "data publishing request") to the Worker. The data publishing request may include: the data identifier of the data generated by the data producer (hereinafter referred to as "data A"). Of course, the data publishing request may also include: the data block storing data A in the memory of the accelerator card bound to the data producer (hereinafter referred to as "data block b1"), the runtime identifier, the identifier of the accelerator card bound to the data producer, etc. Exemplarily, the runtime identifier can be represented by, but is not limited to, the identifier of the Client configured at runtime. When the process is a data consumer, the process can call the data subscription interface in the Client configured therein to send a named object subscription request (or "data subscription request") to the Worker. The data subscription request may include: the data identifier of the data consumed by the data consumer (hereinafter referred to as "data B"). Of course, the data subscription request may also include: a data block storing data B in the memory of the accelerator card bound to the data consumer (hereinafter referred to as "data block b2"), a runtime identifier, an identifier of the accelerator card bound to the data consumer, etc. Exemplarily, the runtime identifier may be, but is not limited to, represented by the identifier of the Client configured in the runtime. Exemplarily, when a process needs to subscribe to multiple data, the process may subscribe to these multiple data in sequence. For example, a process may send multiple data subscription requests in sequence, and different data subscription requests carry different data identifiers, or a process may send one data subscription request, and different data identifiers are arranged in sequence in the data subscription request.

[0118] After receiving a data publishing request and a data subscription request from a runtime, the Worker can perform data object publishing and subscription pairing through its internal named object publishing and subscription pairing module. Specifically, when the data identifier carried in a data publishing request and the data identifier carried in a data subscription request are consistent, the Worker can treat the runtimes that publish these two requests as a data object publishing and subscription pair. For example, the Worker can perform data object publishing and subscription pairing in the order of the data subscription requests it receives and / or the order of the data identifiers carried in the data subscription requests to ensure that the correct P2P communication sequence is subsequently generated. The Worker can then issue a sequence control module through its internal communication operator and generate a P2P communication sequence based on the first-in-first-out (FIFO) principle, i.e., the two runtimes that successfully pair first will perform P2P communication first to transmit data between the two runtimes. After the data object publishing and subscription pairing between the two runtimes is successful, the Worker can notify the two runtimes of the successful pairing according to the P2P communication sequence it generated, and can issue a communication operator to facilitate P2P communication. In addition, when a communication domain is not established between two runtimes that have successfully paired data object publishing and subscription, the Worker can create and manage the root information of the runtime as the root node through the P2P communication domain within it, so that a communication domain can be created between these runtimes, as well as a communication link, etc., to support the subsequent distribution of communication operators. In addition, when the data flow between the two runtimes is completed, the Worker can clean up the object meta-information related to the data transmitted between the two runtimes (such as data identification, data storage location, etc.). Exemplarily, when the data flow between the runtimes is completed, the runtime can notify the Worker to clean up the information related to the data transmitted between the two. In some embodiments, when the data identification of a source data (the data identification in the data publishing request) is successfully paired with the data identification of another target data (the data identification in the data subscription request), if the other data subscription requests received by the Worker need to subscribe to the same source data, the Worker can return a subscription failure message to the runtime that sent the other data subscription request. For example, after the data identifiers of source data 1 and target data 1 are successfully paired, if the data identifier of target data 2 carried in a subsequent data subscription request is also successfully paired with the data identifier of source data 1, the Worker returns a subscription failure message to the runtime that sends the subsequent data subscription request. At this time, the runtime on the side that sends the data subscription request will no longer process the request and notify the runtime of the result of the data subscription through a Future object.

[0119] Furthermore, when the process acting as a data producer (hereinafter referred to as the "producer process") and the process acting as a data consumer (hereinafter referred to as the "consumer process") receive a message from the Worker indicating that the two can perform P2P communication, the producer process can find out the data block storing the data to be transmitted in the memory of the accelerator card bound to it based on the data identifier of the source data related to the successfully paired data identifier, and trigger a dedicated data sending thread to send a data sending operator to the accelerator card so that the accelerator card executes the data sending operator. During the execution of the data sending operator, the accelerator card can extract the data to be transmitted from the corresponding data block and send the data to the consumer process through the communication domain between the runtime of the producer process and the runtime of the consumer process. In addition, the consumer process can find out the data block storing the data it consumes in the memory of the accelerator card bound to it, and trigger a dedicated data receiving thread to send a data receiving operator to the accelerator card so that the accelerator card executes the data receiving operator. During the execution of the data receiving operator, the accelerator card receives the data produced by the producer process through the communication domain between the runtime of the producer process and the runtime of the consumer process, and stores the data in the corresponding data block.

[0120] exist Figure 3 and 4 The technical concept illustrated primarily enables the dynamic delivery of communication operators and the dynamic creation and pre-warming of communication domains. Dynamic delivery of communication operators utilizes the publish-subscribe mechanism of named objects to express dynamically generated P2P communication relationships. The operator delivery sequence agreed upon by the sender and receiver based on the data object names triggers the dynamic delivery of communication operators on both ends, and object metadata is cleaned up after data transfer is complete. Dynamic creation and pre-warming of communication domains involves the on-demand creation and pre-warming of link and cache communication domains when two runtime communication domains do not exist.

[0121] For easier understanding Figure 3 and 4 The technical concept shown is explained below using two runtimes as examples.

[0122] For example, Figure 5 As shown, during the execution of Runtime 1, after generating new data, the Runtime 1 process can call the data publishing interface provided by its client to send a data publishing request to the Worker through the client. During the execution of Runtime 3, before using the new data, the Runtime 3 process can call the data subscription interface provided by its client to send a data subscription request to the Worker through the client.

[0123] After receiving a data publish request from Runtime 1 and a data subscription request from Runtime 3, the Worker can use the data identifiers contained in these requests to pair the data objects. Assuming the data identifiers of the data produced by Runtime 1 and the data required by Runtime 3 are the same, the two can be successfully paired. The Worker can then notify Runtimes 1 and 3 of the communication operator dispatch, in the order in which the data to be transmitted between the two processes is successfully paired. The communication operator dispatched by Runtime 1 is the data sending operator, and the communication operator dispatched by Runtime 3 is the data receiving operator.

[0124] Furthermore, after establishing a communication domain between two runtimes, their processes can cache previously established communication domain information (e.g., domain identifier, the identifier of the other runtime, the thread that created the communication domain, etc.). When a process in runtime 1 or 3 receives a successful pairing response from the Worker, it first checks its stored communication domain list to determine whether a communication domain has already been established with the other end. If so, it directly uses the cached communication domain for operator delivery. If not, the receiving end is assumed to be the root node (although the sending end can also be the root node). The receiving end generates root information and sends it to the Worker. The sending end then sends a request to the Worker for the corresponding receiving end's root information. Using the same root information, the two processes then call the collective communication library interface to establish a communication domain. Once a communication domain is established between the two runtimes, processes in runtimes 1 and 3 can locate the data blocks in the memory of their bound accelerator cards according to the order in which the communication operators are delivered by the Worker, and then execute the communication operator delivery. After the data transfer is complete, the Worker can clean up the object metadata related to the data transferred between the two runtime processes.

[0125] As can be seen from the above description, the core approach to solving the concurrent data flow processing in multiple runtimes in the data migration solution provided by the embodiments of this application is to achieve consistent communication timing between multiple senders and receivers through the collaboration of multiple clients and workers. Furthermore, in principle, for a process in a given runtime, communication is triggered in the order in which data dependencies are agreed upon between the sender and receiver, thereby achieving optimal throughput and avoiding the blocking of other data transmissions due to unfinished dependencies.

[0126] For example, if Figure 6As shown, when a runtime process is a data producer for data A and a data consumer for data B, the runtime process can send a data subscription request and a data publishing request to the Worker by calling the interface in the Client configured within it. After receiving the request from the runtime process, the Worker can record the metadata of the data A published by the runtime and the request information of the data B subscribed to by the runtime in its object record table (Object Directory). For example, the Object Directory can include an object location (ObjectLocations) table and an Object Get request subscription table. The Object Locations table can record the metadata of the data published by the runtime process, such as: the runtime identifier of the published data, the data identifier of the published data, the storage location of the published data (such as which memory data block on which accelerator card, etc.). The Object Get request subscription table can record Get request subscription requests, that is, requests to subscribe to data. The Object Get request subscription table can include the runtime identifier of the subscribed data, the data identifier of the subscribed data, the storage location of the subscribed data (such as which memory data block on which accelerator card, etc.), etc.

[0127] After receiving a data publish request or data subscription request from another runtime process, a Worker can perform a data object publish-subscribe pairing. After a successful data object publish-subscribe pairing, the Worker can trigger a transmission communication operator to send an event. Each time a data object publish-subscribe pairing is completed, the Worker can send an event notification to the runtime process, instructing it to send the corresponding communication operator. An event notification notifies the runtime process that at least one data item has been successfully paired, and the corresponding data send operator or data receive operator can be sent in sequence. Furthermore, the event notification may include the identifier of the other runtime process with which the runtime process is performing data transmission. For example, a Worker can maintain an event subscription table and an event table. The event subscription table can record event query requests from the data producer process, and these requests are misses, meaning that no events were generated. The event table can record events that have been paired. For event notifications, runtime processes can subscribe to or query the Worker. This allows the Worker to immediately notify the runtime process when a new event is generated, enabling instant data transmission between runtime processes. In this embodiment, the sending end (i.e., the end of the process that publishes data) can publish data identifiers by calling the data publishing interface of its runtime client. After the sending end client calls the data publishing interface, the internal implementation of the interface will first call an RPC request to notify the Worker of the release message of the data object identifier. The request returns immediately, which can timely release communication resources. The client then records the release request in a data identifier release table. The runtime call to the data publishing interface is completed and can continue to publish other data. At this time, the internal implementation of the client then asynchronously sends an event query request to the Worker through a send loop thread. This query request can query all events of the data identifiers published by the client to see if they are successfully paired. When an event with a successful event pairing is found, the client queries the data identifier table for the corresponding data identifier release request based on the data identifier and other information carried in the Worker event notification. The request is then handed off to the thread pool dedicated to handling collective communication tasks. When the data transmission is complete, the data identifier data transmission result is notified via a Future. In addition, when the Worker receives data subscription requests from multiple processes simultaneously, the Worker can, but is not limited to, perform parallel matching on the data identifiers of the target data in these data subscription requests.

[0128] After receiving an event notification from a worker, a runtime process can use another runtime identifier included in the event notification to locate the communication domain it needs to use from the communication domain list stored on it and dispatch the corresponding communication operator to the accelerator card where it resides. When a runtime process receives multiple event notifications sequentially, it can dispatch communication operators to the accelerator card in the order of the event notifications. The accelerator card can run a sending thread and a receiving thread. The sending thread executes the data sending operator, and the receiving thread executes the data receiving operator. When a runtime process receives multiple event notifications, it can classify these events to form a sending event queue and a receiving event queue. The sending event queue continues to sort these events in the order in which they were received, and the receiving event queue continues to sort these events in the order in which they were received. In both the sending and receiving event queues, events received earlier are placed at the front of the queue, meaning that the event queues follow a first-in-first-out principle. Furthermore, the process at runtime can sequentially construct the data sending operator according to the order of events in the sending event queue, and sequentially construct the data receiving operator according to the order of events in the receiving event queue, thereby obtaining the data sending operator queue and the data receiving operator queue.

[0129] from Figure 6 As can be seen in the description, regarding communication timing control, Workers use data publish / subscribe to establish a communication operator dispatching sequence between the sender and receiver. After a data object publish / subscribe pairing is successfully established, communication operator events are dispatched to both the sender and receiver in FIFO order. Upon receiving a communication operator dispatch event, the sender and receiver locate the corresponding memory data block and trigger a dedicated send or receive thread to dispatch the data send / receive operator, thus achieving dynamic point-to-point communication. Furthermore, objects can be published and subscribed to simultaneously within the same runtime, meaning data can be published and subscribed to simultaneously.

[0130] To facilitate understanding of the above description, the following further illustrates the above description with examples.

[0131] For example, Figure 7 As shown in the figure, each client configured in the runtime has a sending loop thread and a receiving loop thread, because the client can both publish and subscribe to data. The sending loop thread is responsible for processing data publishing requests, and the receiving loop thread is responsible for processing data subscription requests. Figure 6Client 1 is the sender, and Client 2 is the receiver. For ease of illustration, only the corresponding send and receive loop threads are shown. In reality, both threads can exist simultaneously within each client. A thread pool dedicated to collective communication tasks exists within the client. The number of threads in this thread pool is equal to the total number of communication slots. Different collective communication libraries may have different limits on the total number of communication slots. The number of communication slots refers to the number of concurrent communication domains supported by the accelerator card.

[0132] Continue reading Figure 7 For example, in Step 1: Runtime1's process first publishes a piece of data. Client1 will internally call a PutP2P2Meta remote procedure call protocol (RPC) request to the Worker. If there is only one Worker, then the Worker also serves as the Master for managing metadata. The Worker and the Master are the same process and can be directly accessed through function calls. If there are multiple Workers, another Worker is selected through a hash algorithm to serve as the Master for managing metadata. In this case, the Worker and the Master are different processes. Therefore, the Worker needs to forward the PutP2P2Meta RPC request to the Master.

[0133] After receiving the PutP2P2Meta request, the Master first queries the Object Get request subscription table to determine whether a GetP2Pmeta subscription request exists for the receiving end. If so, the Master responds to the GetP2PMeta request and returns the event that successfully matched the event. Furthermore, the Master queries the Events subscription table and returns the event information that successfully matched the receiving end to Client 1 via the SubsRecvEvent query request from the sender. If no GetP2PMeta subscription request exists, the Master stores the metadata of the data object published by the sender in the Object Locations table.

[0134] For the sender, when the SubsRecvEvent query request successfully finds an event that is successfully matched with the data published by Client1, the sending loop thread selects an idle thread from the thread pool that handles collective communication tasks and hands the event to this thread for processing. The idle thread determines whether the communication domain exists. If not, it creates and caches it; if it exists, it reuses it. It also queries the address of the source data corresponding to the successfully matched data identifier and then sends the data sending operator.

[0135] On the receiving end, when Client2 sends a data subscription request, it places the subscription request in a Sub request queue for management. The receiving loop thread then retrieves each Sub request from the queue and processes it, sending a GetP2PMeta RPC request to the Worker. The Worker forwards the request to the Master or directly calls a function on the same node. Upon receiving the request, the Master first checks the Object Locations table to determine whether the data has been published. If so, it adds a Get event to the Events table and queries the Events subscription table for a matching SubsRecvEvent query request from the sender. It retrieves the corresponding Get event notification from the Events table, responds to the SubsRecvEvent query request, and notifies the sender to send the data. If the data has not been published, it adds a GetP2PMeta RPC request to the ObjectGet request subscription table, waiting for a certain period of time for data to be matched. Upon expiration of this period, it times out and processes the next Sub request. This continues in turn until a match is found or until Client2 exits. Similarly, upon receiving a matching event, the receiving loop thread on the receiving end can hand off processing to the corresponding thread pool.

[0136] exist Figure 7 In the data subscription request, when a data subscription request simultaneously subscribes to multiple data sets (i.e., carries multiple data identifiers), the data identifiers of these data sets can be arranged sequentially in the data subscription request. After receiving the data subscription request, the Worker can sequentially match the data identifiers of these data sets with the data identifiers of the source data recorded on the Worker. Upon completing the data pairing and a successful pairing event occurs, the Worker can return an event notification to the corresponding sender and receiver. The order of the communication operators issued by this event notification is consistent with the order of the data identifiers subscribed to in the data subscription request, thereby achieving P2P communication timing control. Furthermore, for the Worker, after each data subscription request is sent to the worker, it waits for a matching request for the corresponding data identifier within a set timeout period. If a matching request occurs within the timeout period, the data subscription request immediately returns a subscription success event to both the sender and receiver. If no matching request occurs within the timeout period, the data subscription request returns a subscription failure event to the sender's processing thread. The sender can then reinsert the data subscription request into the subscription request queue and continue processing the next subscription request in the subscription request queue.

[0137] from Figure 7As can be seen from the description in , by controlling the order of communication operators at both ends of the transmitter and receiver based on the order of data subscription, the control of communication timing is achieved, the correctness of P2P communication is ensured, and the deadlock problem and disorder problem in P2P communication are solved. Among them, the deadlock problem refers to the situation where two-way communication operators of multiple data are sent to the same operator queue at both ends of fixed P2P communication. The disorder problem refers to the situation where two ends of fixed P2P communication are executed concurrently, and the different order of operator sending at both ends of the transmitter and receiver will lead to data disorder. For example, Figure 8 As shown, when multiple communication domains communicate concurrently with P2P, deadlock may occur due to the timing of communication operator delivery. Figure 7 It can be seen from the description that these problems are solved by the communication timing control scheme in the embodiment of this application.

[0138] In addition, when sending communication operators at runtime, the computing node where the runtime is located can allocate corresponding threads and operator sending queues according to the communication resource slots to assign communication tasks on the communication domain to the corresponding threads for execution. The communication resource slots refer to the number of concurrent communication domains supported by the accelerator card where the runtime is located. Figure 9As shown, the mapping between threads and operator dispatching can be as follows: threads correspond one-to-one to operator dispatching queues. A queue can handle multiple communication domains, and the same communication domain can only be in one queue. Furthermore, the total number of threads equals the total number of communication slots. This ensures thread control on the compute node, avoiding unnecessary thread startup and resolving communication errors caused by exceeding communication resource limits. Furthermore, because communication domains cannot be reused, the same communication domain is in the same queue. Of course, a queue can have different communication domains. For example, due to the constraints of the collective pass library, each client is bound to an accelerator card. Assume that each accelerator card can only support eight communication domains processing operators concurrently. This means that a client may cache n communication domains internally, but only eight can transmit data simultaneously. If more than eight are available, an error will be generated. Suppose client 0 is the sender and publishes 10 data items, key1, key2, ..., key10. Then, there are 10 clients as receivers, each receiving one data item. For client 0, 10 data transmissions could occur simultaneously, meaning 10 communication domains are concurrently transmitting data. This would exceed the communication domain resource limit and result in an error. To address this issue, this embodiment provides a solution: based on the number of communication resource slots, the client internally allocates a corresponding number of task processing threads, with each thread managing a task queue. For example, thread 1, thread 2, ..., thread n (n is the number of communication resource slots). Continuing with the above example, with n = 8, when client 0 sends data to clients 1 through 10, there are 10 send tasks. Since there are only 8 worker threads, the excess tasks must be queued. For example, threads 1-7 are each assigned only one send task, while thread 8 is assigned three. Thread 1's queues are {Task 1}, ..., thread 7's queue is {Task 7}, and thread 8's queue is {Task 8, Task 9, Task 10}. If multiple tasks exist in thread 8, they must be queued and processed one by one. This ensures that only eight communication domains are transmitting at a time, thus resolving the error issue that can be caused by communication resource limitations.

[0139] In addition, when communication is required between two runtimes and no communication domain is created, such as Figure 10 As shown in (A), due to the constraints of the collective communication library, the establishment of communication domain 2 must be after the execution of step 3, and the creation of the communication domain must complete one data transmission and reception before the next one can be created. Therefore, before step 3 is successfully executed, communication domain 2 cannot be created successfully. In addition, since a thread can only use the communication domain created by itself, that is, the communication domain cannot be reused between threads, thread 2 cannot use communication domain 1 to send and receive data in step 5. In view of this, a communication link establishment concurrency control design scheme is proposed in the embodiment of the present application, such as Figure 10 As shown in (B), in this solution, the corresponding concurrency conflict problem is solved by serializing the creation of multiple communication domains and DMA link preheating during runtime, and it can support data flow under runtime expansion and contraction. Among them, concurrency control includes two key designs: 1) The creation of a communication domain during the same runtime requires obtaining the Root node information, and the next RootInfo (i.e., Root node information) is obtained after the communication domain is successfully initialized; 2) The first data transmission and reception during the same runtime will trigger the DMA link initialization, and the next link establishment will be carried out after the DMA link establishment is completed. Continue to refer to Figure 10 (B), the creation of communication domain 1 and communication domain 2 needs to be serialized: first obtain the root information of communication domain 1, create communication domain 1, and perform DMA link preheating on communication domain 1. Among them, DMA link preheating means sending a small data first to complete the initialization of the channel. The data sent by DMA link preheating is irrelevant to the data subsequently transmitted through communication domain 1. Furthermore, communication domain 2 can be created in parallel during or before the subsequent use of communication domain 1 to transmit the corresponding data. For example, in Figure 10 In (B), the task queue in the runtime can be the task queue that the thread needs to execute. For example, the task queue executed by the sending thread a is the task sent by each data sending operator. The communication domain list in the runtime is the communication domain related to the tasks in the task queue. For example, if runtime 1 needs to send data to runtime 2, the communication domain list in the sending thread a can include the communication domain between runtimes 1 and 2. Exemplarily, when establishing a communication domain between two runtimes, Worker can assume the role of Root information flow between runtimes.

[0140] It should be understood that the above solution is not only applied to Figure 3 In the distributed scenario shown, it can also be applied to a single computing node. The specific implementation can be determined according to the actual situation and is not limited here. The process applied to a single computing node can also be referred to the above description and will not be repeated here.

[0141] The specific implementation process of the above technical concept is described below.

[0142] For example, Figure 11 The flowchart of a data migration method provided by an embodiment of the present application is shown. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Exemplarily, the method can be executed by a data migration device, wherein the device can be implemented by software and / or hardware, and can be, but is not limited to, configured in the aforementioned computing node. Figure 11 As shown, the data migration method may include the following steps:

[0143] S1101. Receive a first data subscription request sent from at least one first process, wherein one first data subscription request carries a data identifier of at least one target data, one first process runs on a first accelerator card, one target data is data to be read by the first process, and target data related to the same first data subscription request is read by the same first process.

[0144] In this embodiment, the server can receive first data subscription requests sent by each first process. Each first data subscription request carries a data identifier of at least one target data. A first process can run on a first accelerator card, and a target data is data to be read by a first process, and the data is not stored on the first accelerator card on which the first process runs. In addition, the target data related to the same first data subscription request is read by the same first process. Exemplarily, the first process can call a data subscription interface to send a first data subscription request to the server.

[0145] S1102: Pair the data identifiers of each target data with the data identifiers of the source data recorded in the server, wherein one source data is stored on a second accelerator card and generated by a second process, and one second process runs on the second accelerator card.

[0146] In this embodiment, the server can pair the data identifiers of each target data carried in each first data subscription request with the data identifiers of the source data obtained from at least one second process recorded thereon. The second process is a process that generates source data. One source data can be generated by one second process. One second process runs on a second accelerator card, and the source data generated by the second process is stored in the memory of the second accelerator card on which it runs. In some embodiments, when a first data subscription request carries data identifiers of multiple target data, the server can pair them one by one with the data identifiers of the source data recorded thereon according to the order of these data identifiers, so as to ensure the correctness of the order in which subsequent communication operators are issued. In addition, if the data identifier carried by any first data subscription request is not paired successfully, the server can continue to wait for the data identifier carried by the first data subscription request to be paired within the set pairing timeout period until the pairing is successful or the pairing timeout period is reached, thereby avoiding frequent communication with the corresponding first and saving communication resources.

[0147] S1103. When the data identifier of at least one target data carried in the second data subscription request is successfully matched with the data identifier of the source data recorded in the server, an event notification is sent to the third process and at least one fourth process; wherein the second data subscription request is any first data subscription request received by the server, the third process is the process that sends the second data subscription request in at least one first process, the fourth process is the process that generates the first source data, and the data identifier of the first source data is the data identifier of the source data that is successfully matched with the data identifier of the target data in the second data subscription request; the event notification is used to notify the third process and the fourth process to send a communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process.

[0148] In this embodiment, the second data subscription request can be any first data subscription request received by the server. At this time, if the data identifier of at least one target data carried in the second data subscription request is successfully matched with the data identifier of the source data recorded in the server, the server can respectively send an event notification to the first process and the second process related to the successfully matched data identifier. For ease of description, the "first process related to the successfully matched data identifier" will be referred to as the "third process", the "second process related to the successfully matched data identifier" will be referred to as the "fourth process", the "source data related to the successfully matched data identifier" will be referred to as the "first source data", and the "source data not related to the successfully matched data identifier" will still be referred to as the "source data". The name of the "source data not related to the successfully matched data identifier" can be determined according to actual circumstances and is not limited here. Among them, the event notification is used to notify the third process and the fourth process to send a communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process.

[0149] Exemplarily, the communication operator may include a data sending operator and a data receiving operator. Among them, the event of notifying the sending of the data receiving operator may be referred to as a "first event notification". The event of notifying the sending of the data sending operator may be referred to as a "second event notification" or a "third event notification". At this time, when the server sends the first event notification to the third process, it may be to notify the third process to send at least one data receiving operator to the first accelerator card running the third process. A data receiving operator can be used to receive a first source data from the second accelerator card side running a fourth process. In addition, when there are multiple first source data, the order in which these multiple data receiving operators are sent may be the same as the order in which the first source data is paired. That is to say, the order of the corresponding data receiving operators is arranged according to the order of the data identifiers of the target data in the second data subscription request to achieve correct P2P communication timing.

[0150] When the server matches the target data in the second data subscription request, if multiple first source data are matched, that is, multiple source data are associated with the successfully matched data identifier, and these source data come from different fourth processes, the server can send second event notifications to each fourth process in the order of the matched first source data. In this case, one second event notification can be used to notify one fourth process to send a data sending operator to the first accelerator card running the fourth process, and one data sending operator is used to send one first source data.

[0151] When the server is pairing the target data in the second data subscription request, if there are multiple first source data paired, and these source data come from a fourth process, the server can send a third event notification to the fourth process. At this time, the third event notification is used to notify the fourth process to send multiple data sending operators to the first accelerator card running the fourth process. Among them, one data sending operator is used to send one first source data, and the order in which the data sending operators are sent is the same as the order in which the first source data is paired. In other words, the order of the corresponding data sending operators is arranged according to the order of the data identifiers of the target data in the second data subscription request to achieve correct P2P communication timing.

[0152] As a possible implementation method, before sending an event notification to the third process and the fourth process, the server can also verify whether the size of the data block storing the first target data on the memory of the first accelerator card is the same as the size of the data block storing the source data related to the first target data on the memory of the second accelerator card, and verify whether the number of data blocks storing the first target data on the memory of the first accelerator card is the same as the number of data blocks storing the source data related to the first target data on the memory of the second accelerator card. At this time, the third process is running on the first accelerator card, and the fourth process is running on the second accelerator card. The first target data is any target data subscribed to by the aforementioned second data subscription request. When the verified size and quantity are the same, the server sends an event notification to the third process and the fourth process. This ensures that the source data related to the first target data can be correctly stored on the first accelerator card, avoiding storage errors.

[0153] In addition, after issuing an event notification, the server can clear the source data information associated with the successfully paired data identifier recorded in the event notification, thereby implementing relevant lifecycle management of the source data. In addition, when a process disconnects from the server, the server can also clear the information related to the source data published by the process recorded on it.

[0154] This streamlined approach, through runtime P2P communication based on data publish-subscription, simplifies the representation of data dependencies within the accelerator card's memory, shielding against the constraints and limitations imposed by the underlying hardware and collective communication software libraries. This ensures high throughput while ensuring ease of use. Furthermore, developers no longer need to write complex communication domain management, concurrency control, and order-preserving logic based on the underlying communication libraries, significantly improving the efficiency of intelligent computing development.

[0155] For example, Figure 12 A flow chart of another data migration method provided by an embodiment of the present application is shown. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Exemplarily, the method can be executed by a data migration device, wherein the device can be implemented by software and / or hardware, and can be but not limited to being configured on an accelerator card (for ease of description, it will be referred to as the "first accelerator card" below, and the process on it will be referred to as the "first process") where a process for subscribing to data in the aforementioned computing node is located. As Figure 12 As shown, the data migration method may include the following steps:

[0156] S1201: The first accelerator card calls a data subscription interface through a first process and sends a data subscription request to a server. The data subscription request carries a data identifier of target data, and the target data is data to be read by the first process.

[0157] In this embodiment, when the first process needs to process certain data (or "target data"), and the memory of the first accelerator card running the first process does not store the target data, the first accelerator card can call a data subscription interface through the first process to send a data subscription request to the server. The data subscription request carries the data identifier of the target data.

[0158] S1202. The first accelerator card receives an event notification from the server, wherein the event notification is sent when a data identifier of the target data and a data identifier of at least one source data recorded by the server are successfully matched, and a data identifier of the source data is sent to the server by a second process, and a second process runs on a second accelerator card.

[0159] In this embodiment, the first accelerator card can obtain an event notification from the server. After the server matches the data identifier of the target data in the data subscription request sent by the first process, if there is at least one data identifier of the source data that is the same as the data identifier of the target data in the data identifier of the source data recorded by the server, that is, the pairing is successful, then the server can send an event notification to the first accelerator card. The event notification can be used to notify the first accelerator card that the data identifier pairing is successful and the corresponding communication operator can be sent. The data identifier of a source data recorded by the server is sent to the server by a second process, and a second process is running on a second accelerator card, and the source data is stored on the corresponding second accelerator card.

[0160] S1203: In response to the event notification, the first accelerator card calls the operator delivery interface through the first process to deliver a data receiving operator to the collective communication library on the first accelerator card. The data receiving operator is used to receive the first source data from the memory side of the second accelerator card storing the first source data, and store it in the memory of the first accelerator card. The first source data is source data related to the successfully paired data identifier.

[0161] In this embodiment, after receiving the event notification, the first accelerator card can call the operator sending interface through the first process to send the data receiving operator to the collective communication library on the first accelerator card. The data receiving operator is used to receive the first source data from the memory side of the second accelerator card that stores the first source data, and store it in the memory of the first accelerator card. The first source data is the source data related to the data identifier that is successfully paired. Exemplarily, when the data subscription request sent by the first accelerator card carries multiple data identifiers of target data, and at least two data identifiers of target data are successfully paired with the data identifier of the source data recorded by the server, the event notification can notify the first accelerator card to send the corresponding data receiving operators in sequence according to the order in which the data identifiers are successfully paired. At this time, there are multiple data receiving operators, and the generation order of the data receiving operators is the same as the data identifier order of the target data related to the successfully paired data identifier in the data subscription request. This ensures that the P2P communication timing is correct. In some embodiments, the output parameters of the data receiving operator may include: address information of a free data block in the memory of the first AI accelerator card, and a communication domain established between the runtime environment of the first process and the runtime environment of the second process; the output parameters of the data receiving operator may include: first source data.

[0162] As a possible implementation method, before sending the data receiving operator, the first accelerator card can first query whether there is a first communication domain that can be reused from the communication domain list cached by the first process. The first communication domain is the communication domain established between the runtime environment of the first process and the runtime environment of the third process. The third process is the process that generates the first source data. In other words, first find out whether there is a communication domain required for transmitting data. If there is a reusable first communication domain, the first accelerator card uses the thread that creates the first communication domain in the thread pool in the first process, calls the operator sending interface, and sends the data receiving operator to the collective communication library on the first accelerator card. Exemplarily, the threads in the thread pool can be, but are not limited to, threads specifically responsible for creating communication domains. The number of threads in the thread pool can be the number of concurrent communication domains supported by the first accelerator card, so that the problem of communication resources occupying more than the limit can be solved. For details, please refer to the above description, which will not be repeated here.

[0163] If there is no reusable first communication domain, the first accelerator card uses the idle thread in the thread pool in the first process to create the first communication domain. Then, the first accelerator card uses the thread that creates the first communication domain in the thread pool, calls the operator sending interface, and sends the data receiving operator to the collective communication library on the first accelerator card. Exemplarily, when creating the first communication domain, the first accelerator card can use the idle thread in the thread pool in the first process to send a Root information query request to the server to query the Root information generated by the third process. Then, the first accelerator card can use the thread that sends the Root information query request to receive the Root information returned from the server. Finally, the first accelerator card can use the thread that sends the Root information query request to create the first communication domain based on the Root information. Since the Root information contains information such as the IP address on the side of the third process, the first accelerator card can establish a communication relationship with the device on the side of the third process, thereby establishing the first communication domain.

[0164] After creating the first communication domain, the first accelerator card can cache the first communication domain and record the identifier of the runtime environment of the third process (i.e., the identifier of the first communication domain's peer end), as well as the thread that created the first communication domain within the thread of the first process. This allows for convenient subsequent querying of the existence of reusable communication domains and the ability to call the operator dispatch interface through the corresponding thread.

[0165] In addition, after creating the first communication domain, the first accelerator card can also perform at least one data transmission and reception in the first communication domain, such as receiving data from the other end, to establish a communication link on the first communication domain. The data transmitted by the data transmission and reception at this time is unrelated to the first source data. This is to achieve preheating of the first communication domain. Furthermore, in the process of using the first communication domain to transmit the first source data, the first accelerator card can use the idle threads in the thread pool of the first process in parallel to create a communication domain between the runtime environment of the first process and other runtime environments. This is to achieve concurrent control of communication link establishment.

[0166] In this way, through runtime P2P communication based on data publishing and subscription, the data dependency expression in the accelerator card's memory is simplified, the constraints and limitations brought by the underlying hardware and collective communication software library are shielded, and high throughput is achieved while ensuring ease of use.

[0167] For example, Figure 13 A flow chart of another data migration method provided by an embodiment of the present application is shown. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Exemplarily, the method can be executed by a data migration device, wherein the device can be implemented by software and / or hardware, and can be, but is not limited to, an accelerator card (for ease of description, it will be referred to as the "second accelerator card" below, and the process on it will be referred to as the "second process") where a process that publishes data in the aforementioned computing node is located. As Figure 13 As shown, the data migration method may include the following steps:

[0168] S1301. The second accelerator card calls a data publishing interface through a second process and sends a data publishing request to a server. The data publishing request carries a data identifier of at least one source data generated by the second process.

[0169] In this embodiment, when source data (hereinafter referred to as "first source data") is generated during the execution of the second process, the second accelerator card can call the data publishing interface through the second process and send a data publishing request to the server. The data publishing request carries the data identifier of at least one source data generated by the second process. Exemplarily, the source data generated by the second process can be stored in the memory of the second accelerator card.

[0170] S1302. The second accelerator card receives an event notification from the server. The event notification is sent when a data identifier of the first target data and a data identifier of the first source data recorded by the server are successfully matched. The data identifier of the first target data is sent to the server by the first process.

[0171] In this embodiment, the second accelerator card can obtain an event notification from the server. The server can process-pair the data identifier of the first source data with the data identifier of the target data in the data subscription request sent by the other accelerator card through the corresponding process. When the server successfully matches the data identifier of the first target data with the data identifier of the first source data it has recorded, the server can issue an event notification to the second accelerator card. This event notification can be used to notify the second accelerator card that the data identifier pairing is successful and to issue the corresponding communication operator, etc.

[0172] S1303: In response to the event notification, the second accelerator card calls the operator delivery interface through the second process to deliver a data sending operator to the collective communication library on the second accelerator card, where the data sending operator is used to send the first source data to the memory of the first accelerator card running the first process.

[0173] In this embodiment, after receiving the event notification, the second accelerator card can call the operator sending interface through the second process to send the data sending operator to the collective communication library on the second accelerator card. The data sending operator is used to send the first source data to the memory of the first accelerator card running the first process. The first process is the process that processes the first target data. In some embodiments, the input parameters of the data sending operator may include: the data identifier of the first source data, the address information of the data block storing the first source data in the memory of the second accelerator card, and the communication domain established between the runtime environment of the first process and the runtime environment of the second process; the output parameters of the data sending operator may include: the first source data.

[0174] For example, when the data publishing request sent by the second accelerator card carries the data identifiers of multiple first source data, and at least two data identifiers of the first source data are successfully paired with the data identifiers of multiple target data carried in a data subscription request on the server side, the event notification can notify the second accelerator card to send the corresponding data sending operators in sequence according to the order in which the data identifiers are successfully paired. In this case, there are multiple data sending operators, and the order in which the data sending operators are generated is the same as the order in which the various first source data are paired, and these data sending operators send data to the same first accelerator card. This ensures the correct timing of P2P communication.

[0175] As a possible implementation method, before sending the data sending operator, the second accelerator card can first query whether there is a first communication domain that can be reused from the communication domain list cached by the second process. The first communication domain is the communication domain established between the runtime environment of the second process and the runtime environment of the first process (that is, the process that needs to process the first target data). In other words, first find out whether there is a communication domain required for transmitting data. If there is a reusable first communication domain, the second accelerator card uses the thread that creates the first communication domain in the thread pool in the second process, calls the operator sending interface, and sends the data sending operator to the collective communication library on the second accelerator card. Exemplarily, the threads in the thread pool can be, but are not limited to, threads specifically responsible for creating communication domains. The number of threads in the thread pool can be the number of concurrent communication domains supported by the second accelerator card, so that the problem of communication resources occupying more than the limit that may occur can be solved. For details, please refer to the above description, which will not be repeated here.

[0176] If there is no reusable first communication domain, the second accelerator card uses the idle thread in the thread pool in the second process to create the first communication domain. Then, the second accelerator card uses the thread that creates the first communication domain in the thread pool, calls the operator sending interface, and sends the data sending operator to the collective communication library on the second accelerator card. Exemplarily, when creating the first communication domain, the second accelerator card can use the idle thread in the thread pool in the second process to send a Root information query request to the server to query the Root information generated by the first process (i.e., the process that needs to process the first target data). Then, the second accelerator card can use the thread that sends the Root information query request to receive the Root information returned from the server. Finally, the second accelerator card can use the thread that sends the Root information query request to create the first communication domain based on the Root information. Since the Root information contains information such as the IP address on the first process side, the second accelerator card can establish a communication relationship with the device on the first process side, thereby establishing the first communication domain.

[0177] After creating the first communication domain, the second accelerator card can cache the first communication domain and record the identifier of the runtime environment of the first process (i.e., the identifier of the first communication domain's peer), as well as the thread that created the first communication domain within the thread of the second process. This allows for convenient subsequent querying of the existence of reusable communication domains and the ability to call the operator dispatch interface through the corresponding thread.

[0178] In addition, after creating the first communication domain, the second accelerator card can also perform at least one data transmission and reception in the first communication domain, such as receiving data from the other end, to establish a communication link on the first communication domain. The data transmitted by the data transmission and reception at this time is unrelated to the first source data. This is to achieve preheating of the first communication domain. Furthermore, in the process of using the first communication domain to transmit and receive the first source data, the second accelerator card can use the idle threads in the thread pool of the second process in parallel to create a communication domain between the runtime environment of the second process and other runtime environments. This is to achieve concurrent control of communication link establishment.

[0179] In this way, through runtime P2P communication based on data publishing and subscription, the data dependency expression in the accelerator card's memory is simplified, the constraints and limitations brought by the underlying hardware and collective communication software library are shielded, and high throughput is achieved while ensuring ease of use.

[0180] The above Figures 11 to 13 Combined with the described content, the above-mentioned data migration method mainly simplifies the dependency expression of data in the memory of the accelerator card through runtime P2P communication based on data publishing and subscription. Furthermore, when a data sending operator is issued on one accelerator card and a data receiving operator is issued on another accelerator card, and these two operators are used to transmit the same data, the two accelerator cards can run the corresponding operators on each of them, thereby directly transferring the data in the memory of one accelerator card to the memory of another accelerator card, greatly improving the data transmission efficiency. That is to say, in this embodiment, after the publish-subscribe pairing is successful, the data in the memory of one accelerator card can be directly transferred to the memory of another accelerator card.

[0181] In addition, for the communication domain for transmitting data between the sending end and the receiving end, it is possible to consider using the communication domain in one direction to reduce the probability of data blocking.

[0182] In some embodiments, when processes are bound to accelerator cards at runtime, due to constraints in the collective communication library, multiple processes running on a single card can create conflicting communication domains, leading to errors. Since the interface provided in this embodiment uses the collective communication library for card-to-card data transmission, to avoid communication domain conflicts, different processes at runtime will be bound to different accelerator cards when using the interface provided in this embodiment.

[0183] It is understandable that the size of the serial number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application. In addition, the various embodiments described above can be combined according to actual conditions, and the combined solution is still within the scope of protection of this application. In addition, the Serverless computing platform described above can also be replaced by other platforms, for example, by a computing cluster containing multiple computing nodes, etc., and the replaced solution is still within the scope of protection of this application.

[0184] Next, based on the method in the above embodiment, the data migration device provided in the embodiment of the present application is introduced.

[0185] For example, Figure 14 FIG1 shows a schematic diagram of the structure of a data migration device provided by an embodiment of the present application. For example, the device can be but is not limited to being deployed on the aforementioned server. Figure 14 As shown, the data migration device 1400 includes: a communication module 1401 and a processing module 1402. The communication module 1401 is used to receive a first data subscription request sent from at least one first process, wherein a first data subscription request carries a data identifier of at least one target data, a first process runs on a first accelerator card, a target data is data to be read by a first process, and the target data related to the same first data subscription request is read by the same first process. The processing module 1402 is used to pair the data identifier of each target data with the data identifier of the source data recorded in the server, wherein a source data is stored on a second accelerator card and generated by a second process, and a second process runs on the second accelerator card. The communication module 1401 is also used to send an event notification to the third process and at least one fourth process when the data identifier of at least one target data carried in the second data subscription request is successfully matched with the data identifier of the source data recorded in the server. Among them, the second data subscription request is any first data subscription request received by the server, the third process is the process that sends the second data subscription request in at least one first process, the fourth process is the process that generates the first source data, and the data identifier of the first source data is the data identifier of the source data that is successfully paired with the data identifier of the target data in the second data subscription request; the event notification is used to notify the third process and the fourth process to send the communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process.

[0186] In some embodiments, the processing module 1402 sends an event notification to the third process, including: sending a first event notification to the third process, wherein the first event notification is used to notify the third process to send at least one data receiving operator to the first accelerator card running the third process; wherein one data receiving operator is used to receive one first source data, and when there are multiple first source data, the order in which the data receiving operators are sent is the same as the order in which the first source data are paired.

[0187] In some embodiments, the processing module 1402 sends an event notification to at least one fourth process, including: when there are multiple paired first source data and they come from different fourth processes, sending a second event notification to each fourth process in sequence according to the order of the paired first source data, wherein one second event notification is used to notify one fourth process to send a first data sending operator to the first accelerator card running the fourth process, wherein one first data sending operator is used to send one first source data; when there are multiple paired first source data and they come from one fourth process, sending a third event notification to the fourth process, wherein the third event notification is used to notify the fourth process to send multiple second data sending operators to the first accelerator card running the fourth process, wherein one second data sending operator is used to send one first source data, and the sending order of the second data sending operators is the same as the order of the paired first source data.

[0188] In some embodiments, the communication module 1401 is further used to: when a communication domain is not established between the runtime environment of the third process and the runtime environment of the fourth process, receive first root information generated from the third process, and send the first root information to the fourth process, wherein the first root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process; or, when a communication domain is not established between the runtime environment of the third process and the runtime environment of the fourth process, receive second root information generated from the fourth process, and send the second root information to the third process, wherein the second root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process.

[0189] In some embodiments, the processing module 1402 is further used to: if the data identifier carried by the third data subscription request is not successfully paired, continue to wait for pairing of the data identifier carried by the third data subscription request within a set pairing timeout period until the pairing is successful or the pairing timeout period is reached, wherein the third data subscription request is any first data subscription request received by the server.

[0190] In some embodiments, before sending event notifications to the third process and the fourth process respectively, the processing module 1402 is also used to: determine that the size of the data block storing the first target data on the first memory is the same as the size of the data block storing the source data related to the first target data on the second memory, and the number of data blocks storing the first target data on the first memory is the same as the number of data blocks storing the source data related to the first target data on the second memory, wherein the first memory is the memory of the first accelerator card running the third process, the second memory is the memory of the second accelerator card running the fourth process, and the first target data is any target data subscribed to by the second data subscription request.

[0191] In some embodiments, after sending the event notification to the third process and the fourth process respectively, the processing module 1402 is further configured to: clear the source data information related to the data identifier of the successful pairing recorded by the server.

[0192] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0193] For example, Figure 15 FIG2 shows a schematic diagram of the structure of a data migration device provided by an embodiment of the present application. For example, the device can be but is not limited to being deployed on the aforementioned first accelerator card, on which a first process is running. Figure 15 As shown, the data migration device 1500 includes a communication module 1501 and a processing module 1502. The communication module 1501 is configured to, through a first process, call a data subscription interface to send a data subscription request to a server. The data subscription request carries a data identifier of at least one target data item, which is the data to be read by the first process. The communication module 1501 is also configured to receive an event notification from the server. The event notification is sent when the data identifier of the target data item is successfully matched with the data identifier of at least one source data item recorded by the server. The data identifier of the source data item is sent to the server by a second process running on a second accelerator card. The processing module 1502 is configured to, in response to the event notification, call an operator delivery interface through the first process to deliver at least one data receiving operator to the collective communication library on the first accelerator card. The data receiving operator is configured to receive first source data from the memory of the second accelerator card storing the first source data and store it in the memory of the first accelerator card. The first source data item is the source data associated with the successfully matched data identifier.

[0194] In some embodiments, the generation order of multiple data receiving operators is the same as the order of data identifiers successfully paired in the data subscription request.

[0195] In some embodiments, when the processing module 1502 calls the operator sending interface through the first process and sends the data receiving operator to the collective communication library on the first accelerator card, it is specifically used to: in the presence of a reusable first communication domain, use the thread of the first communication domain created in the thread pool in the first process, call the operator sending interface, and send the data receiving operator to the collective communication library on the first accelerator card, wherein the first communication domain is the communication domain established between the runtime environment of the first process and the runtime environment of the third process, and the third process is the process that generates the first source data.

[0196] In some embodiments, when the processing module 1502 calls the operator sending interface through the first process and sends the data receiving operator to the collective communication library on the first accelerator card, it is specifically used to: in the absence of a reusable first communication domain, use the idle thread in the thread pool in the first process to create the first communication domain, wherein the first communication domain is the communication domain established between the runtime environment of the first process and the runtime environment of the third process, and the third process is the process that generates the first source data; use the thread that creates the first communication domain in the thread pool to call the operator sending interface and send the data receiving operator to the collective communication library on the first accelerator card.

[0197] In some embodiments, when the processing module 1502 uses an idle thread in a thread pool in a first process to create a first communication domain, it is specifically used to: use the idle thread to send a root information query request to the server, wherein the root information query request is used to query the root information generated by a third process, wherein the third process is the process that generates the first source data; use the idle thread to receive the root information returned from the server; and use the idle thread to create the first communication domain based on the root information.

[0198] In some embodiments, the processing module 1502 is further configured to: cache the first communication domain, record the identifier of the runtime environment of the third process, and record the thread that created the first communication domain.

[0199] In some embodiments, after the processing module 1502 uses the idle thread in the thread pool in the first process to create the first communication domain, it is also used to: perform at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link on the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

[0200] In some embodiments, the processing module 1502 is further configured to: in the process of using the first communication domain to transmit the first source data, use idle threads in the thread pool in parallel to create a communication domain between the runtime environment of the first process and other runtime environments.

[0201] In some embodiments, the number of threads in the thread pool is the number of concurrent communication domains supported by the first accelerator card.

[0202] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0203] For example, Figure 16 The schematic diagram of the structure of a data migration device provided by an embodiment of the present application is shown. For example, the device can be but is not limited to being deployed on the aforementioned second accelerator card, on which a second process is running, and the data generated by the second process includes at least one first source data, which is stored in the memory of the second accelerator card. Figure 16 As shown, the data migration device 1600 includes: a communication module 1601 and a processing module 1602. The communication module 1601 is used to receive an event notification from the server, wherein the event notification is sent when the data identifier of the first target data and the data identifier of the first source data recorded by the server are successfully paired, and the data identifier of the first target data is sent to the server by the first process. The processing module 1602 is used to respond to the event notification, call the operator sending interface through the second process, and send at least one data sending operator to the collective communication library on the second accelerator card, wherein the data sending operator is used to send the first source data to the memory of the first accelerator card running the first process.

[0204] In some embodiments, one data sending operator is used to send one first source data, and multiple data sending operators send data to the same first accelerator card, wherein the generation order of the data sending operators is the same as the order in which the first source data is paired.

[0205] In some embodiments, when the processing module 1602 calls the operator through the second process and sends the data sending operator to the collective communication library on the second accelerator card, it is specifically used to: in the presence of a reusable first communication domain, use the thread of the first communication domain created in the thread pool in the second process, call the operator sending interface, and send the data sending operator to the collective communication library on the second accelerator card, wherein the first communication domain is the communication domain established between the runtime environment of the first process and the runtime environment of the second process.

[0206] In some embodiments, when the processing module 1602 calls the operator through the second process and sends the data sending operator to the collective communication library on the second accelerator card, it is specifically used to: in the absence of a reusable first communication domain, use the idle thread in the thread pool in the second process to create a first communication domain, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the second process; use the thread that creates the first communication domain in the thread pool to call the operator sending interface and send the data sending operator to the collective communication library on the second accelerator card.

[0207] In some embodiments, when the processing module 1602 uses an idle thread in a thread pool in a second process to create a first communication domain, it is specifically used to: use the idle thread to send a root information query request to the server, wherein the root information query request is used to query the root information generated by the first process; use the idle thread to receive the root information returned from the server; and use the idle thread to create the first communication domain based on the root information.

[0208] In some embodiments, after the processing module 1602 uses an idle thread in the thread pool to create the first communication domain, it is also used to: perform at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link on the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

[0209] In some embodiments, the processing module 1602 is further configured to: in the process of transmitting the first source data using the first communication domain, use idle threads in the thread pool in parallel to create a communication domain between the runtime environment of the second process and other runtime environments.

[0210] In some embodiments, the processing module 1602 is further configured to cache the first communication domain, record an identifier of the runtime environment of the first process, and record a thread that creates the first communication domain.

[0211] In some embodiments, the number of threads in the thread pool is the number of concurrent communication domains supported by the second AI accelerator card.

[0212] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0213] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, which, when executed by a computing node, causes the computing node to execute the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive), etc.

[0214] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing node, causes the computing node to execute the method in the above embodiment.

[0215] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0216] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0217] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0218] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data migration method, characterized in that: Applied to the server, the method includes: receiving a first data subscription request sent by at least one first process, wherein one first data subscription request carries a data identifier of at least one target data, one first process runs on a first accelerator card, one target data is data to be read by the first process, and target data related to the same first data subscription request is read by the same first process; Pairing data identifiers of each target data with data identifiers of source data recorded in the server, wherein one source data is stored on a second accelerator card and generated by a second process, and one second process is running on the second accelerator card; When the data identifier of at least one target data carried in the second data subscription request is successfully matched with the data identifier of the source data recorded in the server, an event notification is sent to the third process and at least one fourth process; The second data subscription request is any one of the first data subscription requests received by the server, the third process is a process that sends the second data subscription request in the at least one first process, the fourth process is a process that generates the first source data, and the data identifier of the first source data is the data identifier of the source data that is successfully matched with the data identifier of the target data in the second data subscription request; The event notification is used to notify the third process and the fourth process to send a communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process.

2. The method according to claim 1, characterized in that The sending of the event notification to the third process includes: Sending a first event notification to the third process, wherein the first event notification is used to notify the third process to send at least one data receiving operator to the first accelerator card running the third process; Among them, one data receiving operator is used to receive one first source data, and when there are multiple first source data, the sending order of the data receiving operator is the same as the order of matching the first source data.

3. The method according to claim 1 or 2, characterized in that The sending of the event notification to the at least one fourth process includes: If there are multiple paired first source data and they come from different fourth processes, a second event notification is sent to each fourth process in sequence according to the order of the paired first source data, wherein one second event notification is used to notify one fourth process to send a first data sending operator to the first accelerator card running the fourth process, wherein one first data sending operator is used to send one first source data; In the case that there are multiple paired first source data and they come from one fourth process, a third event notification is sent to the fourth process, wherein the third event notification is used to notify the fourth process to send multiple second data sending operators to the first accelerator card running the fourth process, one second data sending operator is used to send one first source data, and the sending order of the second data sending operators is the same as the order in which the first source data are paired.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In a case where a communication domain is not established between the runtime environment of the third process and the runtime environment of the fourth process, receiving first root information generated by the third process, and sending the first root information to the fourth process, wherein the first root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process; or, In a case where a communication domain is not established between the runtime environment of the third process and the runtime environment of the fourth process, second Root information generated by the fourth process is received, and the second Root information is sent to the third process, wherein the second Root information is used to establish a communication domain between the runtime environment of the third process and the runtime environment of the fourth process.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: If the data identifier carried by the third data subscription request is not paired successfully, continue to wait for pairing with the data identifier carried by the third data subscription request within a set pairing timeout period until pairing is successful or the pairing timeout period is reached, wherein the third data subscription request is any one of the first data subscription requests received by the server.

6. The method according to any one of claims 1 to 5, characterized in that: Before sending the event notification to the third process and the fourth process respectively, the method further includes: Determine that the size of a data block storing the first target data on the first memory is the same as the size of a data block storing source data related to the first target data on the second memory, and the number of data blocks storing the first target data on the first memory is the same as the number of data blocks storing source data related to the first target data on the second memory, wherein the first memory is the memory of the first accelerator card running the third process, the second memory is the memory of the second accelerator card running the fourth process, and the first target data is any target data subscribed to by the second data subscription request.

7. The method according to any one of claims 1 to 6, characterized in that: After sending the event notification to the third process and the fourth process respectively, the method further includes: Clear the source data information related to the successfully paired data identifier recorded by the server.

8. A data migration method, characterized in that: Applied to a first accelerator card, where a first process is running on the first accelerator card, the method includes: Calling a data subscription interface through the first process to send a data subscription request to the server, wherein the data subscription request carries a data identifier of at least one target data, and the target data is the data to be read by the first process; receiving an event notification from the server, wherein the event notification is sent when a data identifier of the target data is successfully matched with a data identifier of at least one source data recorded by the server, and the data identifier of the source data is sent to the server by a second process, and the second process runs on a second accelerator card; In response to the event notification, an operator delivery interface is called by the first process to deliver at least one data receiving operator to the collective communication library on the first accelerator card, wherein the data receiving operator is used to receive the first source data from the memory side of the second accelerator card storing the first source data, and store it in the memory of the first accelerator card, where the first source data is source data related to the data identifier that has been successfully paired.

9. The method according to claim 8, characterized in that The generation order of the multiple data receiving operators is the same as the order of the successfully paired data identifiers in the data subscription request.

10. The method according to claim 8 or 9, characterized in that The calling the operator delivery interface by the first process to deliver the data receiving operator to the collective communication library on the first accelerator card includes: In the presence of a reusable first communication domain, a thread of the first communication domain is created in the thread pool in the first process, the operator delivery interface is called, and the data receiving operator is delivered to the collective communication library on the first accelerator card, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of a third process, and the third process is the process that generates the first source data.

11. The method according to claim 8 or 9, characterized in that The calling the operator delivery interface by the first process to deliver the data receiving operator to the collective communication library on the first accelerator card includes: In the absence of a reusable first communication domain, creating the first communication domain using an idle thread in a thread pool in the first process, wherein the first communication domain is a communication domain established between a runtime environment of the first process and a runtime environment of a third process, and the third process is a process that generates the first source data; The thread of the first communication domain is created in the thread pool, the operator delivery interface is called, and the data receiving operator is delivered to the collective communication library on the first accelerator card.

12. The method according to claim 11, characterized in that The step of creating the first communication domain by using an idle thread in a thread pool in the first process includes: Using the idle thread to send a root information query request to the server, wherein the root information query request is used to query root information generated by a third process, wherein the third process is the process that generates the first source data; Using the idle thread to receive the Root information returned from the server; The first communication domain is created based on the Root information using the idle thread.

13. The method according to claim 11 or 12, characterized in that The method further comprises: The first communication domain is cached, and the identifier of the runtime environment of the third process is recorded, as well as the thread that creates the first communication domain.

14. The method according to any one of claims 11 to 13, characterized in that: After creating the first communication domain using an idle thread in a thread pool in the first process, the method further includes: Perform at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link in the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

15. The method according to claim 14, characterized in that The method further comprises: During the process of using the first communication domain to transmit the first source data, idle threads in the thread pool are used in parallel to create a communication domain between the runtime environment of the first process and other runtime environments.

16. The method according to any one of claims 11 to 15, characterized in that: The number of threads in the thread pool is the number of concurrent communication domains supported by the first accelerator card.

17. A data migration method, characterized in that: Applied to a second accelerator card, a second process is running on the second accelerator card, data generated by the second process includes at least one first source data, and the first source data is stored in a memory of the second accelerator card; The method comprises: receiving an event notification from the server, wherein the event notification is sent when a data identifier of first target data and a data identifier of the first source data recorded by the server are successfully matched, and the data identifier of the first target data is sent to the server by the first process; In response to the event notification, the operator sending interface is called by the second process to send at least one data sending operator to the collective communication library on the second accelerator card, wherein the data sending operator is used to send the first source data to the memory of the first accelerator card running the first process.

18. The method according to claim 17, characterized in that One data sending operator is used to send one first source data, and multiple data sending operators send data to the same first accelerator card, wherein the generation order of the multiple data sending operators is the same as the order of pairing the first source data.

19. The method according to claim 17 or 18, characterized in that The calling of the operator by the second process to send the data sending operator to the collective communication library on the second accelerator card includes: In the presence of a reusable first communication domain, a thread of the first communication domain is created in the thread pool in the second process, an operator delivery interface is called, and the data sending operator is delivered to the collective communication library on the second accelerator card, wherein the first communication domain is a communication domain established between the runtime environment of the first process and the runtime environment of the second process.

20. The method according to claim 17 or 18, characterized in that The calling the operator delivery interface by the second process to deliver the data sending operator to the collective communication library on the second accelerator card includes: In the absence of a reusable first communication domain, creating the first communication domain using an idle thread in a thread pool in the second process, wherein the first communication domain is a communication domain established between a runtime environment of the first process and a runtime environment of the second process; Use the thread of the first communication domain created in the thread pool to call the operator sending interface to send the data sending operator to the collective communication library on the second accelerator card.

21. The method according to claim 20, characterized in that The step of creating the first communication domain by using an idle thread in a thread pool in the second process includes: Using the idle thread to send a root information query request to the server, wherein the root information query request is used to query the root information generated by the first process; Using the idle thread to receive the Root information returned from the server; The first communication domain is created based on the Root information using the idle thread.

22. The method according to claim 20 or 21, characterized in that After creating the first communication domain using an idle thread in a thread pool, the method further includes: Perform at least one data transmission and reception in the first communication domain, wherein the data transmission and reception is used to establish a communication link in the first communication domain, and the data transmitted by the data transmission and reception is unrelated to the first source data.

23. The method according to claim 22, characterized in that The method further comprises: During the process of using the first communication domain to transmit the first source data, idle threads in the thread pool are used in parallel to create a communication domain between the runtime environment of the second process and other runtime environments.

24. The method according to any one of claims 20 to 23, characterized in that: The method further comprises: The first communication domain is cached, and an identifier of the runtime environment of the first process is recorded, as well as a thread that creates the first communication domain.

25. The method according to any one of claims 19 to 24, characterized in that: The number of threads in the thread pool is the number of concurrent communication domains supported by the second AI accelerator card.

26. A data migration device, characterized in that: Deployed on the server, the device includes: a communication module, configured to receive a first data subscription request sent from at least one first process, wherein each first data subscription request carries a data identifier of at least one target data, each first process runs on a first accelerator card, each target data is data to be read by the first process, and target data associated with the same first data subscription request is read by the same first process; a processing module, configured to pair data identifiers of each target data with data identifiers of source data recorded in the server, wherein one source data is stored on a second accelerator card and generated by a second process, and one second process is running on the second accelerator card; The communication module is further configured to send an event notification to the third process and at least one fourth process when a data identifier of at least one target data carried in the second data subscription request is successfully matched with a data identifier of the source data recorded in the server; The second data subscription request is any one of the first data subscription requests received by the server, the third process is a process that sends the second data subscription request in the at least one first process, the fourth process is a process that generates the first source data, and the data identifier of the first source data is the data identifier of the source data that is successfully matched with the data identifier of the target data in the second data subscription request; The event notification is used to notify the third process and the fourth process to send a communication operator, and the communication operator is used to transfer the first source data from the memory of the second accelerator card running the fourth process to the memory of the first accelerator card running the third process.

27. A data migration device, characterized in that: The device is deployed on a first accelerator card, where a first process runs on the first accelerator card, and includes: a communication module, configured to call a data subscription interface through a first process and send a data subscription request to a server, wherein the data subscription request carries a data identifier of at least one target data, and the target data is data to be read by the first process; The communication module is further configured to receive an event notification from the server, wherein the event notification is sent when a data identifier of the target data is successfully paired with a data identifier of at least one source data recorded by the server, and the data identifier of the source data is sent to the server by a second process, and the second process runs on a second accelerator card; A processing module is used to respond to the event notification, call an operator sending interface through a first process, and send at least one data receiving operator to the collective communication library on the first accelerator card, wherein the data receiving operator is used to receive the first source data from the memory side of the second accelerator card storing the first source data, and store it in the memory of the first accelerator card, where the first source data is source data related to the data identifier that is successfully paired.

28. A data migration device, characterized in that: Deployed on a second accelerator card, a second process runs on the second accelerator card, data generated by the second process includes at least one first source data, and the first source data is stored in the memory of the second accelerator card; The device comprises: a communication module, configured to receive an event notification from a server, wherein the event notification is sent when a data identifier of first target data and a data identifier of the first source data recorded by the server are successfully matched, and the data identifier of the first target data is sent to the server by the first process; A processing module is used to respond to the event notification, call the operator sending interface through the second process, and send at least one data sending operator to the collective communication library on the second accelerator card, wherein the data sending operator is used to send the first source data to the memory of the first accelerator card running the first process.

29. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by a computing node, cause the computing node to perform the method according to any one of claims 1 to 25.

30. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing node, the computing node executes the method according to any one of claims 1 to 25.

Images