Information determination method and device for cross-domain model training, equipment, medium and program

Abstract

The invention discloses an information determination method and device for cross-domain model training, equipment, a medium and a program, which are applied to the technical field of computer application, and the method comprises the following steps: obtaining a model segmentation result of a to-be-trained model, and obtaining report information of containers under different model training clusters; determining a container serial number and a master control container address of each container according to the reported information and the model segmentation result; and feeding back the container serial number and the main control container address to the corresponding container of each model training cluster. According to the embodiment of the invention, dynamic allocation of the container serial numbers in the model training process can be realized, the flexibility of determining the container serial numbers can be improved, the model training task migration difficulty can be reduced, and the resource utilization rate can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and in particular to a cross-domain model training information determination method, device, equipment, medium and program. BACKGROUND

[0002] In a large model training scenario, a container serial number needs to be allocated to each node participating in training, so as to assist large model training through the container serial number. A common container serial number allocation method is static allocation. A container serial number is allocated to a node participating in training in a configuration manner before large model training. The traditional static container serial number allocation method has the problems of low resource utilization and poor task scheduling flexibility. Large model training can dynamically adjust computing resources participating in training according to different training stages. The static allocation of the container serial number makes the resource allocation fixed before training. When some nodes participating in training have light computing tasks, the resources corresponding to the nodes cannot be utilized by other nodes in time, causing resource waste. When the task volume suddenly increases, the resources cannot be flexibly obtained from other idle nodes, resulting in a slowdown of model training. In a scheduling scenario, training tasks can be dynamically migrated between different nodes according to real-time computing states, data distribution and other factors. The static container serial number makes the binding relationship between the tasks and the nodes too fixed, making it difficult to realize task migration and reallocation, limiting the flexibility of task scheduling and the optimization space. SUMMARY

[0003] The present application provides a cross-domain model training information determination method, device, equipment, medium and program to solve the problem of fixed allocation of container serial numbers in the model training process, improve the flexibility of container serial number determination, reduce the difficulty of training task migration, and improve resource utilization.

[0004] According to an aspect of the present application, a cross-domain model training information determination method is provided, wherein the method comprises:

[0005] Obtaining a model segmentation result of a to-be-trained model, and obtaining reported information of containers under different model training clusters;

[0006] Determining a container serial number and a master container address of each container according to the reported information and the model segmentation result;

[0007] Feeding back the container serial number and the master container address to the corresponding containers of each model training cluster.

[0008] According to another aspect of the present application, a cross-domain model training information determination device is provided, wherein the device comprises:

[0009] The reporting obtaining module is configured to obtain model split results of a to-be-trained model and obtain reporting information of containers under different model training clusters;

[0010] The information distribution module is configured to determine a container serial number and a master container address of each container according to the reporting information and the model split results.

[0011] The information feedback module is configured to feed back the container serial number and the master container address to corresponding containers of each model training cluster.

[0012] According to another aspect of the present application, an electronic device is provided, which comprises:

[0013] at least one processor; and

[0014] a memory connected to the at least one processor in communication; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the information determination method for cross-domain model training according to any embodiment of the present application.

[0016] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the information determination method for cross-domain model training according to any embodiment of the present application when executed by the processor.

[0017] The technical solution of the embodiments of the present application obtains model split results of a to-be-trained model and reporting information of container nodes under each model training cluster, determines a container serial number and a master control container address of each container according to the reporting information and the model split results, feeds back the container serial number and the master control container address to corresponding containers, thereby dynamically allocating a container serial number to a container through the model split results and the reporting information of each container, matching the container serial number with the performance of the container node, improving the resource utilization rate in the model training process, assisting the data communication between container nodes based on the master control container address, reducing the task migration difficulty, and improving the model training speed.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0020] Figure 1 is a flow chart of a cross-domain model training information determination method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of another cross-domain model training information determination method according to an embodiment of the present application;

[0022] Figure 3 is a flow chart of another cross-domain model training information determination method according to an embodiment of the present application;

[0023] Figure 4 is an architecture schematic diagram of a cross-domain model training information determination method according to an embodiment of the present application;

[0024] Figure 5 is an example diagram of a cross-domain model training information determination method according to an embodiment of the present application;

[0025] Figure 6 is a structural schematic diagram of a cross-domain model training information determination device according to an embodiment of the present application;

[0026] Figure 7 is a structural schematic diagram of an electronic device implementing a cross-domain model training information determination method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the technical personnel in the art better understand the present application scheme, the following will combine the drawings in the embodiments of the present application, and the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and above-described accompanying drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment one

[0030] Figure 1 is a flowchart of an information determination method for cross-domain model training according to embodiment one of the present application. The present embodiment can be applicable to the case of container rank (RANK) allocation in the scenario of large model training across clusters. The method can be performed by an information determination apparatus for cross-domain model training, which can be realized in the form of hardware and / or software. The apparatus can be configured in a control node of a server cluster or a cloud server, which is responsible for the allocation and management of container ranks. As shown in Figure 1 the method comprises:

[0031] Step 110, obtaining a model splitting result of a to-be-trained model, and obtaining reported information of containers under different model training clusters.

[0032] The to-be-trained model can be a large model that needs to be trained in at least two model training clusters. The training task of the to-be-trained model can be split, and the splitting result of the training task of the to-be-trained model can be a model splitting result. The model splitting result can include the task number to be executed by each container node for executing the model training task, the tenant to which it belongs, the container rank range of each model training cluster, and the inter-container communication quality requirement, etc. The reported information can be the information uploaded by each container in each model training cluster for participating in model training. The reported information can include the model training task allocated to it, the network address of the container node, and the information of the model training cluster to which it belongs, etc.

[0033] In the embodiment of the present application, the model split result of the to-be-trained model can be obtained, which can include the allocation of the model training task and the allocation range of the container serial number of the model training cluster, etc. In some embodiments, the generation process of the model split result of the to-be-trained model can be performed by the computing power scheduler of the model training cluster. The computing power scheduler can allocate the model training task of the to-be-trained model and determine the allocation range of the container serial number of each model training cluster. The computing power scheduler can send the generated model split result. At the same time, the control node can also receive the reporting information sent by the container under each model training cluster. The reporting information can be actively reported by each container. The reporting information is used to request to obtain the container serial number, so as to perform the model training task of the to-be-trained model. For example, after the container obtains the model training task allocated by the computing power scheduler, the container can send the reporting information to the control node, so that the control node allocates the container serial number for the container.

[0034] Step 120, determining the container serial number and the master container address of each container according to the reporting information and the model split result.

[0035] Specifically, the container serial number of each container can be allocated according to the reporting information and the model split result, and the master control container can be determined for each container, and the master container address of the master control container can be obtained. The method of allocating the container serial number of each container according to the reporting information and the model split result can include: the reporting information can be sorted by the arrival time, the earlier the reporting information arrives, the smaller the container serial number selected for the container according to the model split result, or the container serial number of the container can be allocated according to the model split result and / or the task amount of the model training task carried by the reporting information. The higher the task amount is, the smaller the container serial number of the corresponding container is. The master control container can include the container that sends the reporting information earliest among the containers that send the reporting information, or the pre-configured container, or the container with the highest resource among the containers that send the reporting information, etc.

[0036] Step 130, feeding back the container serial number and the master container address to the corresponding container of each model training cluster.

[0037] In the embodiment of the present application, the container serial number and the master container address corresponding to each container can be sent to the container, so that the container obtains the allocated container serial number and the master container address.

[0038] The embodiment of the application obtains the model segmentation result of the to-be-trained model and the reporting information of the container nodes under each model training cluster, determines the container serial number and the master control container address of each container according to the reporting information and the model segmentation result, and feeds back the container serial number and the master control container address to the corresponding container, so that the container serial number is dynamically allocated to the container according to the model segmentation result and the reporting information of each container, the container serial number is matched with the performance of the container node, the resource utilization rate in the model training process can be improved, the data communication of the container node is assisted based on the master control container address, the task migration difficulty can be reduced, and the model training speed can be improved.

[0039] On the basis of the above-mentioned embodiment of the application, the container intercommunication demand of each container is determined, and the container intercommunication demand is saved to the network scheduler; wherein the container intercommunication demand includes at least one of the following: cluster container communication demand and inter-cluster container communication demand.

[0040] The container intercommunication demand can be the demand of data interaction between the container and other containers, and the container intercommunication demand can be divided into cluster container communication demand and inter-cluster container communication demand according to whether the interaction crosses the model training cluster, the cluster container communication demand can refer to the data interaction between the containers belonging to the same model training cluster, and the inter-cluster container communication demand can refer to the data interaction between the containers belonging to at least different model training clusters.

[0041] In the embodiment of the application, the communication interaction information between the container to which the container serial number is allocated and other containers can be determined, if the container and the other containers belong to the same model training cluster, an identification information of the cluster container communication demand can be set for the container and the other containers, if the container and the other containers belong to different model training clusters, an identification information of the inter-cluster container communication demand can be set for the container and the other containers, and the cluster container communication demand or the inter-cluster container communication demand can be uploaded to the network scheduler, so that the network scheduler schedules the communication between the containers according to the cluster container communication demand and the inter-cluster container communication demand.

[0042] In some other embodiments of the application, determining the container serial number and the master control container address of each container according to the reporting information and the model segmentation result includes:

[0043] Determine the model training cluster to which the reporting information belongs according to the model training cluster identifier included in each reporting information; extract the cluster container serial number range corresponding to each model training cluster in the model segmentation result; for each model training cluster, sequentially allocate the container serial number in the cluster container serial number range to the container corresponding to the reporting information, and determine the master control container address of each container.

[0044] Embodiment two

[0045] Figure 2 is another flowchart of the information determination method for cross-domain model training provided according to Embodiment Two of the present application, and the present embodiment specifically describes the allocation process of the container serial number, see Figure 2 The method provided by the present embodiment specifically includes the following steps:

[0046] Step 210: receiving a model splitting result generated by the computing power scheduler in splitting the model training task of the to-be-trained model, wherein the model splitting result at least includes a cluster container serial number range.

[0047] The model training task can be an execution task for training the to-be-trained model, the model training task can be executed in a parallel mode, the model training task can be divided into multiple model splitting results, the model splitting result can be generated by splitting the model training task in a data parallel, model parallel or pipeline parallel manner, the model splitting result can be information for controlling the container to execute the model training task, and the model splitting result can at least include the cluster container serial number range of each model training cluster. The cluster container serial number range can be a serial number range of the container serial numbers that the container in the model training cluster can obtain.

[0048] In the present embodiment, the computing power scheduler can divide the model training task of the to-be-trained model into subtasks executed by different containers in a data parallel, model parallel or pipeline parallel manner, and can determine the cluster container serial number range of each model training cluster according to the division of the subtasks and the resource status of the model training cluster. For example, the higher the resource of the model training cluster is, the larger the cluster container serial number range allocated by the model training cluster is. The cluster container serial number range corresponding to each model training cluster can be encapsulated as a model splitting result, and the model splitting result can be sent by the computing power scheduler to the control node for managing and allocating the container serial number, so that the control node obtains the model splitting result. It can be understood that the model splitting result can also include other information, including but not limited to the cluster tenant, the task number of the model training task, the communication quality requirement, etc.

[0049] Step 220: receiving the reporting information sent by different containers in each model training cluster, wherein the reporting information includes the model training cluster identifier and the network address of the container.

[0050] Specifically, the to-be-trained model can be trained by multiple model training clusters, the underlying architecture and resource allocation manner of different model training clusters can be different, multiple containers, for example, PODs in K8S, can be configured under each model training cluster, and before the model training task execution of each POD is allocated, the reporting information is sent to the control node responsible for container serial number management and allocation, the reporting information at least carries the model training cluster identifier and network address of the container, the model training cluster identifier can represent the model training cluster to which the container belongs, and the network address can be the Internet protocol address configured by the container.

[0051] Step 230, determine the model training cluster to which the reporting information belongs according to the model training cluster identifier included in each reporting information.

[0052] Specifically, the model training cluster identifier can be extracted in each reporting information, and each reporting information can be classified according to the model training cluster identifier, so as to determine one or more reporting information corresponding to each model training cluster respectively. It can be understood that the model training cluster identifiers of the reporting information belonging to the same model training cluster can be the same.

[0053] Step 240, determine the information arrival order of each reporting information corresponding to each model training cluster.

[0054] This step is a specific example, and step 240 can be omitted in some embodiments of the application.

[0055] Among them, the information arrival order can be determined according to the time when each reporting information is received, and can be obtained by arranging the time when each reporting information is received.

[0056] Specifically, for each model training cluster, a group of reporting information corresponding thereto is obtained, the model training cluster identifiers carried in the group of reporting information can be the same, the receiving time of each reporting information in the group of reporting information can be obtained, and the reporting information or the receiving time of the reporting information is arranged according to the order of the receiving time, so as to obtain the information arrival order of each model training cluster.

[0057] Step 250, extract the cluster container serial number range corresponding to each model training cluster in the model splitting result.

[0058] In the embodiment of the present application, the model splitting result can carry the cluster container serial number range of each model training cluster, each cluster container serial number range can indicate the value range of the container serial number that can be allocated by the model training cluster container, the number of container serial numbers in the cluster container serial number range corresponding to each model training cluster can be different, the cluster container serial number range can be allocated according to the resource quantity of the model training cluster corresponding thereto, and the more the available resources of the model training cluster, the larger the cluster container serial number range corresponding thereto can be. When allocating the container serial number, the cluster container serial number range of each model training cluster can be extracted in the model splitting result.

[0059] In the embodiment of the present application, the container serial number in the cluster container serial number range corresponding to each model training cluster can be allocated to the container corresponding to the reported information in the order of information arrival. The container serial number allocation mode can be in the order of information arrival, that is, the container serial number in the cluster container serial number range corresponding to the model training cluster can be allocated to the container corresponding to the reported information in the order of information arrival, and the container serial number in the cluster container serial number range corresponding to the model training cluster can be allocated to the container corresponding to the reported information in the order of information arrival.

[0060] This step is only an example, but is not limited to allocating the container serial number to the container corresponding to the reported information in the order of information arrival, for example, the container serial number in the cluster container serial number range in the model training cluster can also be randomly allocated to the container sending the reported information in the model training cluster.

[0061] Step 270, obtaining the earliest arrived target reported information in each reported information, and determining the network address carried by the target reported information as the master container address of each container.

[0062] In the embodiment of the present application, the reported information of each container that can be received can determine the target reported information received earliest in the reported information, can determine the container corresponding to the target reported information as the master container, and can extract the network address carried by the target reported information as the master container address of the master container corresponding to each container. It can be understood that the master container can be the container corresponding to the reported information received earliest among all containers sending the reported information.

[0063] The step is only an example, but is not limited to determining the network address carried by the target reporting information that arrives earliest as the master container address of each container. For example, a network address carried by a randomly selected reporting information in the obtained target reporting information can also be determined as the master container address.

[0064] Step 280, feedback container sequence number and master container address to the corresponding container of each model training cluster.

[0065] In the embodiment of the application, the model splitting result sent by the computing power scheduler is received, the model splitting result is obtained by the computing power scheduler by splitting the model training task of the to-be-trained model, the reporting information sent by each model training cluster container is received, the reporting information is classified according to the model training cluster identifier carried by the reporting information, the corresponding reporting information of each model training cluster is obtained, the information arrival order of the reporting information is determined for each model training cluster, the cluster container sequence number range of the model training cluster is extracted in the model splitting result, the container sequence number in the cluster container sequence number range is sequentially assigned to the container of each reporting information according to the information arrival order in the reporting information corresponding to each model training cluster, the target reporting information that arrives earliest in all reporting information is determined, and the network address carried by the target reporting information is determined as the master container address of the container. The container sequence number and the master control container address are fed back to the corresponding container. In the embodiment of the application, the container sequence number is dynamically assigned to the container by the model splitting result and the reporting information of each container, so that the container sequence number is matched with the container node performance, the resource utilization rate in the model training process is improved, the data communication of the container node is assisted based on the master container address, the task migration difficulty is reduced, and the model training speed is improved.

[0066] Further, on the basis of the above-mentioned embodiment of the application, the container sequence number in the cluster container sequence number range is sequentially assigned to the container corresponding to each reporting information according to the information arrival order, including:

[0067] For the cluster container sequence number range, an association relationship between the container sequence number and the network address carried by the reporting information is established, and the association relationship is saved as the sequence number allocation record of the container.

[0068] Specifically, the currently processed reporting information is determined according to the information arrival order; for the cluster container sequence number range, the container sequence number of the number of network addresses carried by the reporting information is sequentially selected in ascending order; an association relationship between the container sequence number and the network address carried by the reporting information is established, and the association relationship is saved as the sequence number allocation record of the container. The application does not limit the above-mentioned specific order, and the above-mentioned embodiment is only an example.

[0069] In the embodiment of the present application, the reported information corresponding to the model training cluster can be selected as the currently processed reported information in turn according to the information arrival order. For the reported information, the container serial numbers in the range of the cluster container serial numbers can be selected in ascending order of the serial number values. The number of the selected container serial numbers can be determined by the number of network addresses carried by the reported information. For example, if the reported information carries two network addresses, the two container serial numbers in the range of the cluster container serial numbers and with the smallest values can be selected as the container serial numbers of the containers corresponding to the currently processed reported information. The selected container serial numbers can be associated with the network addresses carried by the reported information, so as to generate the association relationship between the container serial numbers and the network addresses. The association relationship can be saved as a serial number allocation record, so as to manage the container serial numbers in the subsequent process. Further, when the reported information carries multiple network addresses, the multiple container serial numbers selected according to the above method can be associated with the network addresses in ascending order of the corresponding network card numbers. That is, the network address with a smaller network card number can correspond to the container serial number with a smaller value.

[0070] Embodiment three

[0071] Figure 3 is a flowchart of another information determination method for cross-domain model training provided by the embodiment three of the present application. The embodiment of the present application describes the feedback process of the container serial numbers. Referring to Figure 3 The method provided by the embodiment of the present application specifically includes the following steps:

[0072] Step 310: obtaining the model segmentation result of the model to be trained, and obtaining the reported information of the containers under different model training clusters.

[0073] Step 320: determining the container serial number and the master container address of each container according to the reported information and the model segmentation result.

[0074] Step 330: distributing the container serial number and the master container address allocated for each container to the container.

[0075] In the embodiment of the present application, the container serial number allocated for each container and the determined master container address can be fed back to the corresponding container under each model training cluster.

[0076] Step 340: controlling the container to report the network address and the container serial number to the master container corresponding to the master container address according to the master container address.

[0077] Specifically, after the container obtains the container serial number and the master container address, the container can be controlled to send its network address and the container serial number to the master container corresponding to the master container address, so that the master container obtains the container serial number and the corresponding network address of each container.

[0078] Step 350, controlling the master container to notify each container of the network address and the container serial number of the other containers.

[0079] In the embodiments of the present application, the master container can be controlled to send the network address and the container serial number of the other containers to each container, so as to realize data communication between the containers.

[0080] In some embodiments of the present application, the container-to-container communication requirement of each container is determined, and the container-to-container communication requirement is saved to the network scheduler, including:

[0081] The container serial number allocated to each container is extracted, and it is determined whether the container serial numbers of any two containers belong to the cluster container serial number range of the same model training cluster. If yes, it is determined that the container-to-container communication requirement of the two containers is the cluster container communication requirement, and if not, it is determined that the container-to-container communication requirement of the two containers is the inter-cluster container communication requirement. The container-to-container communication requirements and the network addresses of the containers are transmitted to the network scheduler, so that the network scheduler schedules the transmission of model training data between the containers.

[0082] In the embodiments of the present application, the container serial number of each container can be obtained, and it is determined whether the container serial numbers of any two containers belong to the container serial number range of a model training cluster. If yes, the container-to-container communication requirement of the above two containers can be marked as the cluster container communication requirement, otherwise, the container-to-container communication requirement of the above two containers can be marked as the inter-cluster container communication requirement. All the obtained container-to-container communication requirements can be uploaded to the network scheduler, so that the network scheduler can schedule the transmission of model training data between different containers according to the above container-to-container communication requirements.

[0083] Embodiment four

[0084] Figure 4 is an architecture schematic diagram of a cross-domain model training information determination method according to the fourth embodiment of the present application, referring to Figure 4When the large model is trained by multiple model training clusters, the container serial numbers can be allocated and managed by a global RANK management module. The RANK management module receives a model segmentation result from a computing power scheduler, and the model segmentation result can include a tenant, a task number, a RANK range of each cluster, and quality requirements for inter-RANK communication, etc. RANK numbers are allocated to all containers according to the model segmentation result, and a MASTER is selected. The result including the RANK numbers and the IP address of the MASTER is notified to all containers. Then, the quality requirements for inter-RANK communication are converted into quality requirements for inter-IP communication according to the correspondence between the RANK numbers and the IP addresses of the containers, and the quality requirements for inter-IP communication are notified to a network scheduler.

[0085] The first RANK number segment is 0 to N in the cluster, the first POD reporting information allocates RANK number 0, and the POD is taken as the MASTER; the second POD reporting information allocates RANK number 1, and so on, and the POD reporting information is allocated the corresponding RANK number.

[0086] The first POD reporting information in the other RANK number segment is N+1 to N+M in the cluster, the second POD reporting information allocates RANK number N+1, and so on, and the POD reporting information in the cluster is allocated the RANK number.

[0087] Further, when the POD has multiple network cards, that is, the POD has multiple IPs, the network cards can be sorted according to the network card names, and the first IP is allocated the smallest RANK number according to the network card name sorting.

[0088] In some embodiments of the application, there can be a cross-domain communication relationship between the last RANK number in the first RANK number segment and the second RANK number segment, and the one-to-one communication relationship can be determined according to the network card name sorting.

[0089] In an exemplary embodiment, taking a large model training task requiring three PODs as an example, the computing power scheduler can schedule the above three PDOs to two clusters: cluster 1 and cluster 2 according to the calculation result. Among them, cluster 1 schedules two PODs: POD1 and POD2, and cluster 2 schedules one POD: POD3. Referring to FIG. 2, the cluster 1 and the cluster 2 are scheduled by the computing power scheduler. Figure 5 In the determination process of the RANK number in the embodiment of the application, the following flow can be included:

[0090] (1) The computing power scheduler notifies the scheduling result

[0091] After the computing power scheduler notifies the scheduling result to the RANK manager, the RANK manager maintains the correspondence between the clusters and the RANK numbers of the task:

[0092] Cluster 1, Task 1, RANK number pool: 0, 1;

[0093] Cluster 2, Task 1, RANK number pool: 2.

[0094] (2) POD1 reports information

[0095] After POD1 starts, it reports its information to the RANK manager, which includes cluster 1, task 1, and IP1. The RANK manager allocates the RANK number 0 to POD1 from the RANK number pool and regards it as the MASTER; at this time, the content in the RANK manager is updated as follows:

[0096] Cluster 1, Task 1, RANK number pool: 1, IP1-RANK0;

[0097] Cluster 2, Task 1, RANK number pool: 2.

[0098] After POD1 receives the reply, it determines itself as the MASTER and waits for other PODs to report information.

[0099] (3) POD2 reports information

[0100] After POD2 starts, it reports its information to the RANK manager, which includes cluster 1, task 1, and IP2. The RANK manager allocates the RANK number 1 to POD2 from the RANK number pool; at this time, the content in the RANK manager is updated as follows:

[0101] Cluster 1, Task 1, RANK number pool: IP2-RANK1, IP1-RANK0;

[0102] Cluster 2, Task 1, RANK number pool: 2.

[0103] Subsequently, the RANK number (1) and the address of the MASTER (IP1) are returned to POD2, and POD2 receives the reply and reports information to POD1.

[0104] (4) POD3 reports information

[0105] After POD3 starts, it reports its information to the RANK manager, which includes cluster 2, task 1, and IP3. The RANK manager allocates the RANK number 2 to POD3 from the RANK number pool; at this time, the content in the RANK manager is updated as follows:

[0106] Cluster 1, Task 1, RANK number pool: IP2-RANK1, IP1-RANK0;

[0107] Cluster 2, Task 1, RANK number pool: IP3-RANK2.

[0108] Subsequently, the RANK number (2) and the address (IP1) of the MASTER are returned to the POD3, and the POD3 reports the information to the POD1 after receiving the reply.

[0109] (5) Inform the network scheduler

[0110] The RANK manager informs the network scheduler of the communication requirements within and between clusters.

[0111] Intra-cluster communication requirements: IP1 (RANK0) and IP2 (RANK1)

[0112] Inter-cluster communication requirements: IP2 (RANK1) and IP3 (RANK2)

[0113] (6) Large model training node communication

[0114] After receiving the reply of the RANK manager, the POD2 and the POD3 report their own information to the POD1 respectively; the POD1 informs the two PODs of the correspondence between the IP and the RANK respectively; after obtaining the IP address of the POD3, the POD2 initiates a Remote Direct Memory Access (RDMA) connection request, and sends the training required data through the RDMA connection.

[0115] The RANK management model provided by the embodiment of the application can realize dynamic allocation of the RANK number, improve the efficiency of large model cross-domain training, and reduce the complexity of large model cross-domain training.

[0116] Embodiment five

[0117] Figure 6 is a structural schematic diagram of an information determination device for cross-domain model training provided by the embodiment five of the application, referring to Figure 6 , the device comprises:

[0118] The reporting acquisition module 410 is configured to acquire a model splitting result of a to-be-trained model, and acquire reporting information of containers under different model training clusters.

[0119] The information distribution module 420 is configured to determine a container serial number and a master container address of each container according to the reporting information and the model splitting result.

[0120] The information feedback module 430 is configured to feed back the container serial number and the master container address to corresponding containers of each model training cluster.

[0121] In the embodiment of the present application, the model segmentation result of the to-be-trained model and the reporting information of the container nodes under each model training cluster are obtained by the reporting obtaining module, the information distribution module determines the container serial number and the master control container address of each container according to the reporting information and the model segmentation result, and the information feedback module feeds back the container serial number and the master control container address to the corresponding container. In the embodiment of the present application, the container serial number is dynamically allocated to the container through the model segmentation result and the reporting information of each container, so that the container serial number is matched with the performance of the container node, the resource utilization rate in the model training process can be improved, the data communication of the container nodes is assisted based on the master control container address, the task migration difficulty can be reduced, and the model training speed can be improved.

[0122] In some embodiments of the application, the communication demand reporting module is further configured to determine the inter-container communication demand of each container and save the inter-container communication demand to the network scheduler, wherein the inter-container communication demand includes at least one of the following: cluster container communication demand, inter-cluster container communication demand.

[0123] Further, on the basis of the above-mentioned embodiments of the application, the reporting obtaining module 410 comprises:

[0124] The segmentation obtaining module is configured to receive the model segmentation result generated by the computing power scheduler when the computing power scheduler segments the model training task of the to-be-trained model, wherein the model segmentation result at least includes the cluster container serial number range.

[0125] The information reporting module is configured to receive the reporting information sent by different containers in each model training cluster, wherein the reporting information includes the model training cluster identifier and the network address of the container.

[0126] On the basis of the above-mentioned embodiments of the application, the information distribution module 420 comprises:

[0127] The information classification unit is configured to determine the model training cluster to which the reporting information belongs according to the model training cluster identifier included in each reporting information.

[0128] The serial number range unit is configured to extract the cluster container serial number range corresponding to each model training cluster in the model segmentation result.

[0129] The serial number allocation unit is configured to allocate the container serial numbers in the cluster container serial number range to the containers corresponding to each reporting information in turn for each model training cluster, and determine the master control container address of each container.

[0130] On the basis of the above-mentioned embodiments of the application, the serial number allocation unit is specifically configured to: for the cluster container serial number range, establish an association relationship between the container serial number and the network address carried by the reporting information, and save the association relationship as the serial number allocation record of the container.

[0131] On the basis of the above-mentioned embodiment of the application, the communication demand reporting module is specifically configured to: extract the container serial number allocated to each container, and determine whether the container serial numbers of any two containers belong to the cluster container serial number range of the same model training cluster; if yes, determine that the inter-container communication demand of the two containers is cluster intra-container communication demand, and if not, determine that the inter-container communication demand of the two containers is cluster inter-container communication demand; and transmit the inter-container communication demands and the network addresses of the containers to the network scheduler, so that the network scheduler schedules the model training data transmitted between the containers.

[0132] On the basis of the above-mentioned embodiment of the application, the information feedback module 430 comprises:

[0133] an information sending unit configured to issue the container serial number allocated to each container and the master container address to the containers;

[0134] a reporting control unit configured to control the containers to report the network address and the container serial number to the master container corresponding to the master container address according to the master container address;

[0135] a container notification unit configured to control the master container to notify each container of the network address and the container serial number of the other containers.

[0136] The information determination apparatus for cross-domain model training provided in the embodiments of the application can perform the information determination method for cross-domain model training provided in any of the embodiments of the application, and has the corresponding function modules and beneficial effects of performing the method.

[0137] Embodiment six

[0138] Figure 7 is a structural schematic diagram of an electronic device for implementing the information determination method for cross-domain model training of the embodiments of the application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are merely examples and are not intended to limit the implementations described and / or claimed in this document.

[0139] As Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0141] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the information determination method for cross-domain model training.

[0142] In some embodiments, the information determination method for cross-domain model training can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the information determination method for cross-domain model training described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the information determination method for cross-domain model training by any other appropriate means, such as by means of firmware.

[0143] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0144] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0145] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0147] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0148] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0149] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0150] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A method for determining information during cross-domain model training, characterized in that, The method includes: Obtain the model splitting results of the model to be trained, and obtain the reporting information of containers under different model training clusters; The container number and main control container address of each container are determined based on the reported information and the model segmentation results. The container number and the address of the master control container are fed back to the corresponding container in each of the model training clusters.

2. The method according to claim 1, characterized in that, Also includes: Determine the inter-container communication requirements for each of the containers and save the inter-container communication requirements to the network scheduler; The inter-container communication requirements include at least one of the following: intra-cluster container communication requirements and inter-cluster container communication requirements.

3. The method according to claim 1, characterized in that, The steps of obtaining the model segmentation results of the model to be trained and obtaining the reporting information of containers under different model training clusters include: The model splitting result generated by the computing power scheduler splitting the model training task of the model to be trained is received, wherein the model splitting result includes at least the cluster container sequence number range; The system receives the reporting information sent by different containers within each model training cluster, wherein the reporting information includes the model training cluster identifier and the network address of the container.

4. The method according to claim 1, characterized in that, The step of determining the container number and master control container address of each container based on the reported information and the model segmentation results includes: The model training cluster to which the reported information belongs is determined according to the model training cluster identifier included in each of the reported information; Extract the cluster container index range corresponding to each model training cluster within the model segmentation result; For each model training cluster, the container numbers within the cluster container number range are sequentially assigned to the containers corresponding to the reported information, and the master control container address of each container is determined.

5. The method according to claim 4, characterized in that, The step of sequentially assigning container serial numbers within the cluster container serial number range to the containers corresponding to each reported information includes: For the range of cluster container serial numbers, establish an association between the container serial number and the network address carried in the reported information, and save the association as a serial number allocation record for the container.

6. The method according to claim 2, characterized in that, The step of determining the inter-container communication requirements of each container and saving the inter-container communication requirements to the network scheduler includes: Extract the container number assigned to each container, and determine whether the container numbers of any two containers belong to the same cluster container number range of the model training cluster. If yes, then the inter-container communication requirement of the two containers is determined to be a cluster intra-container communication requirement; otherwise, the inter-container communication requirement of the two containers is determined to be an inter-cluster container communication requirement. The communication requirements between the containers and the network addresses of the containers are transmitted to the network scheduler so that the network scheduler can schedule the model training data transmitted between the containers.

7. The method according to claim 1, characterized in that, The step of feeding back the container number and the master control container address to the corresponding container in each of the model training clusters includes: The container sequence number and the master control container address assigned to each of the containers are sent to the containers; The container is controlled to report its network address and container number to the master container corresponding to the master container address, based on the master container address. The master control container controls each container to notify the other containers of their network addresses and container numbers.

8. An information determination device for cross-domain model training, characterized in that, The device includes: The reporting and acquisition module is used to obtain the model splitting results of the model to be trained, as well as the reporting information of containers under different model training clusters; The information allocation module is used to determine the container number and master control container address of each container based on the reported information and the model segmentation result; The information feedback module is used to feed back the container number and the address of the master control container to the corresponding container in each of the model training clusters.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the information determination method for cross-domain model training as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the information determination method for cross-domain model training as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the information determination method for cross-domain model training according to any one of claims 1-7.

Images