Distributed Training for Container Scheduling for Intelligent Computing

Abstract

The present disclosure provides a distributed training method for container scheduling for intelligent computing, the method including: dividing a target model to obtain a plurality of sub-models; determining at least one computing node for arranging the plurality of sub-models based on the plurality of sub-models, creating a plurality of containers on the at least one computing node, and arranging the plurality of sub-models in the plurality of containers, respectively; executing a model training task using the sample data to train the plurality of sub-models arranged in the plurality of containers; determining a computing node that needs to adjust a container distribution as a target node based on load data of the at least one computing node and a computing time corresponding to each of the plurality of containers; adjusting the container distribution in the target node with an adjustment target that the computing times corresponding to each of the plurality of containers are close to each other; and executing a training task of the target model based on each computing node whose container distribution has been adjusted.

Description

[Technical field]

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, apparatus, storage medium and electronic device for distributed training for container scheduling for intelligent computing. [Background technology]

[0002] With the development of artificial intelligence, the application field of machine learning has progressed from breadth to depth, and the requirements for model training and application have increased. With the significant increase in model size and training data volume, container-based distributed training has become widely used to improve model training efficiency.

[0003] Specifically, a common method for model training is for a server to place submodels divided based on a model in one or more containers, and for multiple containers to share the computing resources of a computing node (such as a GPU) on a physical machine to perform model training. However, during the model training process, the computing resources of each computing node may change dynamically, and since multiple containers share a single physical machine, the performance of a container may be affected by other containers, which may reduce the efficiency of distributed training.

[0004] Therefore, an urgent issue is how to dynamically adjust the distribution of containers placed on compute nodes when training a model so as to bring the training times of sub-models within a compute node closer together and reduce load imbalance between compute nodes. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides a distributed training method, apparatus, storage medium and electronic device for container scheduling for intelligent computing to solve the above problems in the prior art. [Means for solving the problem]

[0006] The technical solutions used in this disclosure are as follows:

[0007] The present disclosure provides a distributed training method for container scheduling for intelligent computing, the method comprising: obtaining sample data and a target model; Dividing the target model into a plurality of sub-models, each of the plurality of sub-models including a portion of a network layer in the target model; determining at least one computing node for arranging the sub-models based on the sub-models, creating a plurality of containers on the at least one computing node, and arranging the sub-models in the plurality of containers, respectively; running a model training task using the sample data to train the sub-models disposed within the containers; acquiring load data of the at least one computing node when executing a model training task, and determining, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container; determining a computation node that requires container distribution adjustment as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers; adjusting a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other; and executing a training task of the target model based on each computing node on which the distribution of containers has been adjusted.

[0008] Optionally, the step of dividing the target model to obtain a plurality of said sub-models comprises: determining a computation time of the target model when performing a model training task; and dividing a network layer included in the target model based on a calculation time of the target model to obtain the plurality of sub-models.

[0009] Optionally, the step of determining, for each of the plurality of containers, a computation time of the sub-model when executing a training task of the sub-model located in the container, comprises: determining training statistics corresponding to the container from a given shared storage system; The method includes a step of determining a calculation time for the sub-model placed within the container when executing a training task for the sub-model placed within the container, based on the start time and end time of executing the training task for the sub-model placed within the container, which are included in the training statistical information. The training statistics stored in the shared storage system are determined based on target logs generated by each computing node of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each computing node of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

[0010] Optionally, the step of determining a computing node that needs to adjust the distribution of containers based on the load data of the at least one computing node and the computing time corresponding to each of the plurality of containers includes: Sorting the plurality of containers in descending order of the calculation time corresponding to each of the plurality of containers to obtain a first sorting result; determining a container that is prior to a predetermined order in the first sorting result as a target container; determining a computing node for which a distribution of containers needs to be adjusted based on the target container and load data of the at least one computing node.

[0011] Optionally, the step of determining a computing node that needs to adjust a distribution of containers based on the target container and load data of the at least one computing node comprises: determining a computing node on which the target container is placed as a first node; determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among other computation nodes; determining the first node and the second node as computation nodes that need to adjust distribution of containers.

[0012] Optionally, the step of determining a computing node that needs to adjust a distribution of containers based on the target container and load data of the at least one computing node comprises: determining, when it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, a calculation node on which a new container to be created is to be placed as a calculation node on which a distribution of containers needs to be adjusted based on load data of the at least one calculation node; The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes: The method includes the steps of creating a new container in the target node, making a copy of model data of a sub-model placed in the target container, and placing the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

[0013] Optionally, the step of determining a computing node on which a new container to be created is to be placed as a computing node on which distribution of containers needs to be adjusted based on load data of the at least one computing node comprises: sorting the computing nodes other than the computing node on which the target container is placed in order of increasing load data of the at least one computing node to obtain a second sorting result; Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result; Among the other computing nodes, for any two sorted adjacent computing nodes, If it is determined that the load difference between the two sorted adjacent computing nodes is not within the predetermined range, the node with the lower load among the two sorted adjacent computing nodes is set as the computing node on which the new container to be created is placed; If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, determining whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

[0014] Optionally, the method further comprises: determining, as a related submodel, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created when it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range; determining a computation node on which the associated sub-model is located as an associated node; measuring a network delay between the relevant node and a computing node other than the relevant node; The method further includes a step of determining, based on the network delay obtained by the measurement, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

[0015] Optionally, the step of determining a computing node that needs to adjust a distribution of containers based on the target container and load data of the at least one computing node comprises: determining a computing node on which the target container is located; a step of determining, when it is determined that a designated container has been placed on the computation node on which the target container has been placed, that the computation node on which the target container has been placed is a computation node on which a distribution of containers needs to be adjusted, and a sub-model placed on the designated container is the same as the sub-model placed on the target container; The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes: The method includes a step of deleting the target container or the designated container from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

[0016] Optionally, the step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes: A step of adjusting the distribution of containers in the target node, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other and that the loads of the at least one calculation node are close to each other. of include.

[0017] The present disclosure provides a distributed training apparatus for container scheduling for intelligent computing, the apparatus comprising: a first acquisition module configured to acquire sample data and a target model; a segmentation module configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model; a first determination module configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively; a first training module configured to perform a model training task using the sample data to train the sub-models disposed within the containers; a second acquisition module configured to acquire load data of the at least one computing node when performing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model arranged in the container when performing a training task of the sub-model as a computation time corresponding to the container; A second determination module configured to determine a computation node that needs to adjust the distribution of containers as a target node based on the load data of the at least one computation node and the computation time corresponding to each of the plurality of containers; an adjustment module configured to adjust a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other; and a second training module configured to execute a training task of the target model based on each computing node on which the distribution of containers has been adjusted.

[0018] Optionally, the splitting module specifically comprises: determining a computation time for the target model when performing a model training task; The network layer included in the target model is divided based on the calculation time of the target model to obtain the plurality of sub-models.

[0019] Optionally, the second acquisition module specifically comprises: determining training statistics corresponding to the container from a given shared storage system; The method is configured to determine the calculation time of the sub-model when executing the training task of the sub-model placed in the container, based on the start time and end time of executing the training task of the sub-model placed in the container, which are included in the training statistical information. The training statistics stored in the shared storage system are determined based on target logs generated by each computing node of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each computing node of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

[0020] Optionally, the second decision module specifically: sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result; In the first sorting result, a container that is located before a predetermined order is set as a target container; The method is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

[0021] Optionally, the second decision module specifically: determining a computing node on which the target container is placed as a first node; determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among other computation nodes; The first node and the second node are configured to determine the first node and the second node as computation nodes that need to adjust a distribution of containers.

[0022] Optionally, the second decision module specifically: When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to other containers exceeds a second set threshold, the calculation node on which the new container to be created is to be placed is determined as a calculation node for which the distribution of containers needs to be adjusted, based on the load data of the at least one calculation node. The adjustment module specifically includes: The method is configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

[0023] Optionally, the second decision module specifically: sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result; According to the second sorting result, sequentially determine whether a load difference between two sorted adjacent computing nodes is within a predetermined range; Among the other computing nodes, for any two sorted adjacent computing nodes, If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed; If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the system is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

[0024] Optionally, the second decision module further comprises: If it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel; determining a computation node on which the related sub-model is located as a related node; Measure a network delay between the relevant node and a computing node other than the relevant node; The network node is configured to determine, based on the measured network delay, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

[0025] Optionally, the second decision module specifically: determining a computing node on which the target container is placed; When it is determined that a designated container has been placed on the computation node on which the target container is placed, the computation node on which the target container is placed is configured to be a computation node for which a distribution of containers needs to be adjusted, and the sub-model placed on the designated container is the same as the sub-model placed on the target container. The adjustment module specifically includes: The target container or the designated container is deleted from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

[0026] Optionally, the adjustment module specifically: The distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other and the loads of the at least one calculation node are close to each other as adjustment targets.

[0027] The present disclosure provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, which, when executed by a processor, performs the above-mentioned distributed training method for container scheduling for intelligent computing.

[0028] The present disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device performing the distributed training method for container scheduling for intelligent computing described above when the processor executes the computer program. Effect of the Invention

[0029] At least one of the above technical solutions used in the present disclosure can achieve the following beneficial effects:

[0030] The distributed training method for container scheduling for intelligent computing provided in the present disclosure divides a target model to obtain sub-models, determines each computing node for placing each sub-model based on each sub-model, creates a container on each computing node, places each sub-model in each container, respectively, executes a model training task using sample data to train the sub-models placed in each container, determines a computing node for which the distribution of containers needs to be adjusted as a target node based on the load data of each computing node and the computing time corresponding to each container, adjusts the distribution of each container in the target node with the adjustment target that the computing times corresponding to the containers on each computing node on which the sub-models are placed are close to each other, and continues executing the training task of the target model.

[0031] As can be seen from the above method, when performing a model training task, first divide the target model into multiple sub-models, then determine each computing node for placing each sub-model, create a container on each computing node, place each sub-model in each of the containers respectively, and complete the training task through each computing node. In the process of model training, the present disclosure measures the load data of each computing node, and dynamically adjusts the distribution of each container on each computing node with the adjustment goal that the calculation time corresponding to the container on each computing node where the sub-model is placed is close, which is favorable to load distribution among each computing node, and can further improve the efficiency of model training. [Brief description of the drawings]

[0032] The accompanying drawings described herein are used to enhance understanding of the present disclosure and constitute a part of the present disclosure, and the exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure and are not intended to constitute an undue limitation of the present disclosure.

[0033] [Figure 1] FIG. 1 is a schematic diagram showing a flow of a distributed training method for container scheduling for intelligent computing provided in an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram showing system relationships provided in an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a schematic diagram illustrating a container arrangement provided in an embodiment of the present disclosure. [Figure 4] FIG. 2 is a schematic diagram showing the configuration of a distributed training device for container scheduling for intelligent computing provided in an embodiment of the present disclosure. [Diagram 5] 1 is a schematic diagram illustrating an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be described below clearly and completely in conjunction with the specific embodiments of the present disclosure and the corresponding accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments that can be obtained by those skilled in the art without creative efforts belong to the protection scope of the present disclosure.

[0035] Below, the technical solutions provided in each embodiment of the present disclosure will be described in detail in conjunction with the accompanying drawings.

[0036] FIG. 1 is a schematic diagram showing a flow of a distributed training method for container scheduling for intelligent computing provided in the present disclosure embodiment, which includes the following steps:

[0037] In S100, sample data and a target model are obtained.

[0038] In S102, the object model is divided to obtain a plurality of sub-models, each of which includes a part of a network layer in the object model.

[0039] With the significant increase in model size and training data volume, large-scale models may not be fully deployed on a single physical machine, and the memory capacity of a single GPU card cannot meet the requirements for large-scale model training. In order to improve the efficiency of model training, the present disclosure provides a distributed training method for container scheduling for intelligent computing, and performs distributed training of models using a multi-machine multi-GPU method.

[0040] The execution subject of the present disclosure may be a system, an electronic device such as a laptop or desktop computer, or a system for executing a model training task (the system may be configured with a device cluster consisting of multiple terminal devices). For convenience of explanation, the distributed training method for container scheduling for intelligent computing provided in the present disclosure will be described below using only a system as an example of the execution subject.

[0041] The system may obtain sample data and a target model, and divide the target model to obtain multiple sub-models, where each sub-model includes a portion of the network layers in the target model.

[0042] In this disclosure, there are various ways in which the system can perform segmentation of the target model.

[0043] Specifically, the system may determine the operation time of the target model when performing a model training task as the operation time of the target model. According to the determined operation time of the target model, the system may divide different network layers included in the target model, with the adjustment goal that the operation times of each sub-model when performing a model training task are close to each other.

[0044] For example, assuming that the target model includes 30 network layers, the system may divide the target model according to the operation time to obtain two sub-models after division, one of which includes the first 10 network layers of the target model and the other sub-model includes the last 20 network layers of the target model. In this case, when the system executes the training task of the two sub-models, the operation time lengths of the two sub-models are close, that is, the difference between the operation times of the two sub-models is within a predetermined range.

[0045] Of course, the system may directly divide the target model based on the number of network layers included in the target model. Assuming that the network layers included in the target model are 30 layers, the system may equally divide the number of network layers of the target model into two sub-models, one of which includes the first 15 network layers of the target model and the other of which includes the last 15 network layers of the target model. The present disclosure does not limit the method of dividing the model.

[0046] In S104, at least one computing node for arranging the sub-models is determined based on the sub-models, a plurality of containers are created on the at least one computing node, and the sub-models are arranged in the containers, respectively.

[0047] The system may determine, based on the sub-models, computation nodes for placing the sub-models, create containers on the computation nodes, and place the sub-models in the containers, respectively.

[0048] For example, assuming that currently five physical machines are jointly completing a model training task and there are two computing nodes (e.g., GPUs) on each physical machine, after dividing the target model into 20 sub-models, the system may create a total of 20 containers on each computing node, and place the 20 sub-models obtained by the division into the 20 containers respectively.

[0049] At S106, a model training task is performed using the sample data to train a plurality of sub-models disposed within the plurality of containers.

[0050] The system may run a model training task using the sample data to train the sub-models placed within each container.

[0051] Specifically, when performing model training for each sub-model, the system may use a log collection framework to collect relevant data during the training process of each sub-model, where the relevant data includes all data generated by each sub-model during the training process and is used to reflect the calculation and execution status of each sub-model on the container.

[0052] Specifically, the system may collect relevant data using a logging method. When each sub-model is trained, the system may log the start and end of model calculation, the start and end of memory access, and other timings as training statistics.

[0053] In order to filter the training statistics from related data, when outputting the log, the system may add information that can uniquely identify the training thread, such as container address information or thread number, to the log content, and may also add keywords to the log content that distinguish it from other log content, such as container-adaptive-adjust.

[0054] When training a model for each sub-model, the system may continuously scan newly generated logs. When the start log is scanned, the system may record the execution time (e.g., calculation, storage access) of the sub-model in this training process and unique identification information such as a thread number, and then continue scanning until the end log is scanned to calculate the execution time of this training process.

[0055] The system may filter the target logs generated for model training based on keywords, determine the execution start time and end time of the sub-model based on the target logs, and transmit and store information such as the execution time and thread number of the sub-model recorded in the target logs to a shared storage system as training statistics information corresponding to the corresponding container of the sub-model.

[0056] Specifically, for each computing node that executes a model training task, each computing node obtains training statistics of each sub-model through the filtered target log. If the number of training statistics does not exceed the predetermined threshold, the system may continue to scan the log until the number of training statistics exceeds the predetermined threshold. At this time, the system transmits the training statistics to a predetermined shared storage system in bulk.

[0057] After each computing node transmits the training statistical information to the shared storage system in bulk, the system may delete the training statistical information held by each computing node and continue to record the training statistical information corresponding to each container until the distributed training is completed.

[0058] Of course, the system may preset a batch transmission time, and may batch transmit the training statistics to the shared storage system when the time when the system last batch transmitted the training statistics to the shared storage system exceeds the preset transmission time. For example, if the preset transmission time is 15 minutes, the system may batch transmit the training statistics to the shared storage system every 15 minutes during the training process of each sub-model.

[0059] That is, each training statistical information stored in the shared storage system is determined based on the target logs generated by each computing node when executing the model training task, the target logs are filtered from the logs generated by each computing node according to predetermined specified keywords, and the training statistical information is written to the shared storage system and deleted from each computing node after accumulating a certain number or after reaching a predetermined time.

[0060] In S108, load data of the at least one computing node when executing a model training task is obtained, and for each container, the calculation time of the sub-model placed in the container when executing a training task of the sub-model is determined as the calculation time corresponding to the container.

[0061] The system may obtain load data of each computing node when executing a model training task, and for each container, determine the computation time of the sub-model placed in the container when executing a training task of the sub-model as the computation time corresponding to the container.

[0062] The system may then analyze the operating status of the containers based on the computing time corresponding to each container and the load data of each computing node, and adjust the distribution of the containers.

[0063] Specifically, the system may read out the training statistics information corresponding to each container from a predetermined shared storage system, and for each container, the system may determine the calculation time of the sub-model when executing the training task of the sub-model disposed in the container according to the start time and the end time of executing the training task of the sub-model disposed in the container, which are included in the training statistics information.

[0064] During the process of model training, the system continuously writes the training statistics information to the shared storage system, and in order to reduce the impact on distributed training, when the system reads the training statistics information corresponding to each container from the shared storage system to analyze the state of the container, the system may obtain the training statistics information generated during the previous training of each sub-model.

[0065] That is, the training iteration order corresponding to the data used by the system to analyze the operating state of the container is one behind the training iteration order of the current model training. For example, assuming the training iteration order of the current model training is i, the training iteration order corresponding to the training statistics information read by the system from the shared storage system is i-1.

[0066] In addition, in order to improve the performance of the shared storage system, the system may also store the training iteration order as one of the keywords in the shared storage system so that training statistical information corresponding to the same iteration order is stored consecutively.

[0067] In S110, a computation node for which a distribution of containers needs to be adjusted is determined as a target node based on the load data of the at least one computation node and the computation time corresponding to each of the plurality of containers.

[0068] In S112, the distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other as an adjustment target.

[0069] After determining the load data of each computing node and the corresponding computing time of each container, the system may determine the computing node that needs to adjust the distribution of containers as the target node.

[0070] Specifically, the system may obtain a first sorting result by sorting the multiple containers in descending order of the calculation time corresponding to each of the containers, and may select one or more containers that are prior to a predetermined rank in the first sorting result as one or more target containers. The system can reflect the operating state of each container with the computing time corresponding to each container, and then determine the computing nodes that need to adjust the distribution of containers based on the computing time corresponding to each container.

[0071] For example, assuming that the specific value of the predetermined ranking is 5, the system may obtain a first sorting result in descending order of the computation time corresponding to each container on each computing node, and then select the first five containers in the sorting result as the target containers.

[0072] Of course, the system may determine the target container in other ways.

[0073] For example, the system may acquire load information for each computing node, and based on the load information for each computing node, determine the computing node with the lowest GPU utilization rate among the computing nodes. If the GPU utilization rate of the computing node is lower than a predetermined threshold, the container with the highest I / O load on the computing node may be selected as the target container.

[0074] After determining the target container, the system may determine the compute nodes that need to adjust the distribution of the co-containers based on the load data of the target container and each compute node.

[0075] Specifically, the system may determine the computing node on which the target container is placed as the first node, and if it is determined based on the load data of the first node that the load of the first node is higher than a first set threshold, determine from other computing nodes as the second node a computing node for placing a portion of the container in the first node.

[0076] The system determines the first node and the second node as computation nodes that require container distribution adjustment, and can migrate the target container in the first node to the second node with the adjustment goal that the calculation times corresponding to the containers in each computation node where the sub-model is placed are close to each other.

[0077] Here, the first set threshold may be set in advance, or may be the average value of the loads of the computing nodes other than the first node.

[0078] For example, based on the load data of the first node, the system determines that the load value of the first node is 20 (the load value is used to represent the level of load, and the load value is positively correlated with the load), and if the first set threshold is 10, or if the average load value of each computing node at this time is 10, the system needs to determine a computing node other than the first node as the second node for placing some of the containers in the first node.

[0079] Specifically, the system may first determine a target container with the highest I / O load on a first node, and then determine a computation node with the lowest I / O load as a second node based on load data of computation nodes other than the first node, and at this time migrate the target container on the first node to the second node to adjust the distribution of containers on each computation node.

[0080] Of course, in the present disclosure, the computing nodes that require the container distribution adjustment may be determined by other methods.

[0081] If the system determines that the difference between the computation time corresponding to the target container and the computation time corresponding to other containers exceeds a second set threshold, the system may determine, based on the load data of each computation node, the computation node on which to place the new container to be created as the computation node that needs to adjust the distribution of the containers.

[0082] Here, the second set threshold may be preset, or may be an average value corresponding to the corresponding calculation time of each container.

[0083] For example, if it is determined that the number of target containers is 1, the calculation time corresponding to the target container is 20 minutes, and the calculation time corresponding to containers other than the target container is 10 minutes, the system determines that the difference between the calculation time corresponding to the target container and the calculation time corresponding to the other containers exceeds a second set threshold (e.g., 5 minutes), and at this time, based on the load data of each calculation node, the system may determine the calculation node on which to place the new container to be created as the target node, which is the calculation node that needs to adjust the distribution of containers.

[0084] In this case, there are various ways in which the system can determine the target node.

[0085] Specifically, the system may sort the computing nodes other than the computing node on which the target container is placed in order of the load data of the computing nodes from smallest to largeest to obtain a second sorting result, and sequentially determine whether the load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result.

[0086] If it is determined that the load difference between any two sorted adjacent computation nodes among the other computation nodes is not within a predetermined range, the system may select the node with the lower load among the two sorted adjacent computation nodes as the computation node on which to place the new container to be created, and if it is determined that the load difference between the two sorted adjacent computation nodes is within the predetermined range, the system may determine whether the load difference between the next two sorted adjacent computation nodes is within a predetermined range until all computation nodes in the second sorted result are traversed or until the computation node on which to place the new container to be created is determined.

[0087] Here, the load data of each computing node can be expressed in terms of GPU utilization, CPU utilization, memory utilization, and storage device bandwidth.

[0088] For example, the system may sort the computing nodes other than the computing node on which the target container is placed in order of the smallest GPU utilization rate of each computing node.

[0089] If it is determined that the difference in GPU usage between any two sorted adjacent computation nodes among the other computation nodes is not within a predetermined range, the node with the lower GPU usage among the two sorted adjacent computation nodes is set as the computation node on which to place the new container to be created, and if it is determined that the load difference between the two sorted adjacent computation nodes is within the predetermined range, it is determined whether the difference in GPU usage between the next two sorted adjacent computation nodes is within a predetermined range until the computation node on which to place the new container to be created is determined.

[0090] In the second sorting result, if the difference in GPU usage between each two sorted adjacent computing nodes is within a predetermined range, the system may re-sort the computing nodes other than the computing node on which the target container is placed in order of the GPU usage of each computing node from smallest to largest, thereby obtaining the second sorting result again.

[0091] At this time, if it is determined that the difference in GPU usage between any two sorted adjacent computation nodes is not within a predetermined range, the node with the lower GPU usage is selected as the computation node in which to place the new container to be created.

[0092] In this manner, the system may sequentially compare the data size of the compute nodes, such as GPU utilization, CPU utilization, memory utilization, and storage device bandwidth, until a compute node on which to place the new container to be created is determined.

[0093] If the system is unable to determine the compute node on which to place the new container to be created by sequentially comparing the data size of the compute nodes, such as the GPU usage, CPU usage, memory usage, and storage device bandwidth, the system may determine the compute node on which to place the new container to be created in another manner.

[0094] Specifically, if the system determines that the load difference between two sorted adjacent computing nodes in the second sorting result is within a predetermined range, the system may determine a submodel that has a network layer dependency with a submodel corresponding to a new container to be created as a related submodel. For example, if the output of a submodel is an input of another submodel, the two submodels can be related submodels.

[0095] The system may also determine a computation node on which a related submodel is placed as the related node. The system may measure a network delay between the related node and another computation node other than the related node, and determine a computation node on which a new container to be created is placed from among the computation nodes other than the related node based on the network delay obtained by the measurement.

[0096] For example, the system may determine a computation node with the smallest network delay between the associated node and the computation node on which to place a new container to be created, or the system may determine an average value of network delay between the associated node and other computation nodes, and determine another computation node with a network delay between the associated node and the computation node that is lower than the average value on which to place a new container to be created.

[0097] After the computation node on which the new container to be created is placed has been determined, the system may create a new container on the target node, make a copy of the model data of the sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment goal that the computation time corresponding to the container on the computation node on which the sub-model is placed is close.

[0098] Additionally, the system may determine the target node in other ways.

[0099] Specifically, the system may first determine the computation node on which the target container is placed, and if it is determined that the designated container is placed on the computation node on which the target container is placed, the computation node on which the target container is placed may be determined as the computation node on which the container distribution needs to be adjusted, where the sub-model placed on the designated container is the same as the sub-model placed on the target container.

[0100] After the target node is determined, the system may delete the target container or a specified container on the computation node on which the target container is placed, with the adjustment goal that the computation time corresponding to the container on the computation node on which the sub-model is placed is close.

[0101] In other words, when multiple containers with the same model parameters of sub-models are simultaneously placed on one physical node (e.g., the same physical machine), the system may delete the containers on the physical node on which the sub-models are placed, with the adjustment goal that the computation times corresponding to the containers on the computation node on which the sub-models are placed are close to each other, and may retain only one container with the same model parameters of the sub-models placed on that physical node.

[0102] In the present disclosure, when adjusting the distribution of each container in a target node, the system adjusts the distribution of each container in the target node with the adjustment target that the calculation times corresponding to the containers in the computation nodes on which the sub-models are placed are similar and the loads of the computation nodes on which the containers are placed are similar.

[0103] In S114, a training task of the target model is executed based on each computing node for which the distribution of containers has been adjusted.

[0104] After the system adjusts the distribution of each container in the target node, the system may continue to execute the training task of the target model using the sample data based on each computing node with the adjusted distribution of containers.

[0105] Note that, before adjusting the distribution of containers in the target node, the system may perform a checkpoint operation on all current containers and save the training information of the current training iteration order.

[0106] Based on each computing node with the adjusted distribution of containers, the system may obtain the previously saved training information through a load operation, and then launch training threads for the sub-models in all containers to continue training each sub-model. Note that the intermediate training variables of the sub-models in the newly created container may be copied from other containers that are the same as the model data of the sub-model.

[0107] In addition, the above contents described in this disclosure are directed to a distributed training method for container scheduling for intelligent computing, taking only a system as an execution subject as an example. However, in reality, the system may be composed of multiple computing nodes, analyzers, and schedulers.

[0108] FIG. 2 is a schematic diagram illustrating the system relationships provided in this disclosure.

[0109] As shown in FIG. 2, while each computing node performs distributed training for each sub-model, each computing node continues to write training statistics information in bulk to the shared storage system.

[0110] Before adjusting the distribution of containers contained in each computing node, the analyzer may read training statistics from the shared storage system to obtain load data of each computing node when executing a model training task, and for each container, determine the computation time of the sub-model placed in the container when executing a training task of the sub-model as the computation time corresponding to the container.

[0111] After the analyzer determines the computing nodes that need to adjust the distribution of containers based on the load data of each computing node and the computing time corresponding to each container, the scheduler may adjust the distribution of containers on each computing node.

[0112] FIG. 3 is a schematic diagram showing the container arrangement provided in the present disclosure.

[0113] As shown in Fig. 3, the scheduler may adjust the distribution of each container in the target node, with the adjustment target being that the computation times corresponding to the containers in each computation node on which the submodel is placed are close to each other. The specific adjustment method has been described in detail in steps S110 to S112.

[0114] After the scheduler adjusts the distribution of the containers, each computing node may continue to execute the learning task of the target model based on the adjusted distribution of the containers of each computing node.

[0115] As can be seen from the above method, when performing a model training task, first divide the target model into multiple sub-models, then determine each computing node for placing each sub-model, create a container on each computing node, place each sub-model in each container respectively, and complete the training task through each computing node. In the process of model training, the present disclosure measures the load data of each computing node, and dynamically adjusts the distribution of each container on each computing node with the adjustment goal that the calculation time corresponding to the container on each computing node where the sub-model is placed is close, which is favorable to load distribution among each computing node, and can further improve the efficiency of model training.

[0116] The above is the method provided in one or more embodiments of the present disclosure, and based on the same idea, the present disclosure further provides a corresponding distributed training apparatus for container scheduling for intelligent computing, as shown in FIG.

[0117] FIG. 4 is a schematic diagram showing a distributed training device for container scheduling for intelligent computing provided in the present disclosure, the device comprising: a first acquisition module 400 configured to acquire sample data and a target model; a segmentation module 402 configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model; a first determination module 404 configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively; a first training module 406 configured to perform a model training task using the sample data to train the sub-models disposed within the containers; a second acquisition module 408 configured to acquire load data of the at least one computing node when performing a model training task, and determine, for each of the plurality of containers, a computing time of the sub-model arranged in the container when performing a training task of the sub-model as a computing time corresponding to the container; A second determination module 410 configured to determine a computing node that needs to adjust the distribution of containers as a target node based on the load data of the at least one computing node and the computing time corresponding to each of the plurality of containers; an adjustment module 412 configured to adjust the distribution of containers in the target node with an adjustment target that the computation times corresponding to the plurality of containers are close to each other; and a second training module 414 configured to execute a training task of the target model based on each computing node on which the distribution of containers has been adjusted.

[0118] Optionally, the splitting module 402 specifically: determining a computation time for the target model when performing a model training task; The network layer included in the target model is divided based on the calculation time of the target model to obtain the plurality of sub-models.

[0119] Optionally, the second acquisition module 408 specifically: determining training statistics corresponding to the container from a given shared storage system; The method is configured to determine the calculation time of the sub-model when executing the training task of the sub-model placed in the container, based on the start time and end time of executing the training task of the sub-model placed in the container, which are included in the training statistical information. The training statistics stored in the shared storage system are determined based on target logs generated by each computing node of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each computing node of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

[0120] Optionally, the second determination module 410 specifically: sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result; In the first sorting result, a container that is located before a predetermined order is set as a target container; The method is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

[0121] Optionally, the second determination module 410 specifically: determining a computing node on which the target container is placed as a first node; determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among other computation nodes; The first node and the second node are configured to determine the first node and the second node as computation nodes that need to adjust a distribution of containers.

[0122] Optionally, the second determination module 410 specifically: When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to other containers exceeds a second set threshold, the calculation node on which the new container to be created is to be placed is determined as a calculation node for which the distribution of containers needs to be adjusted, based on the load data of the at least one calculation node. Specifically, the adjustment module 412: The method is configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

[0123] Optionally, the second determination module 410 specifically: sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result; According to the second sorting result, sequentially determine whether a load difference between two sorted adjacent computing nodes is within a predetermined range; Among the other computing nodes, for any two sorted adjacent computing nodes, If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed; If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the system is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

[0124] Optionally, the second determination module 410 further comprises: If it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel; determining a computation node on which the related sub-model is located as a related node; Measure a network delay between the relevant node and a computing node other than the relevant node; The network node is configured to determine, based on the measured network delay, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

[0125] Optionally, the second determination module 410 specifically: determining a computing node on which the target container is placed; When it is determined that a designated container has been placed on the computation node on which the target container is placed, the computation node on which the target container is placed is configured to be a computation node for which a distribution of containers needs to be adjusted, and the sub-model placed on the designated container is the same as the sub-model placed on the target container. Specifically, the adjustment module 412: The target container or the designated container is deleted from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

[0126] Optionally, the adjustment module 412 specifically: The distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other and the loads of the at least one calculation node are close to each other as adjustment targets.

[0127] The present disclosure further provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, the computer program being used to perform the distributed training method for container scheduling for intelligent computing provided in FIG. 1 above.

[0128] The present disclosure further provides an electronic device as shown in Fig. 5. As shown in Fig. 5, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, an internal memory, and a non-volatile memory, and may of course include other hardware required for operation. The processor loads a corresponding computer program from the non-volatile memory into the internal memory and executes it to implement the distributed training method for container scheduling for intelligent computing described in Fig. 1 above.

[0129] Of course, in addition to realization by software, this disclosure does not exclude other realization methods, such as logical devices or a combination of hardware and software. In other words, the entity that executes the following processing steps is not limited to each logical unit, but may be hardware or a logical device.

[0130] In the 1990s, improvements in a technology could be clearly divided into hardware improvements (such as improvements in circuit structures such as diodes, transistors, and switches) and software improvements (improvements in method flows). However, with the development of technology, many of the current method flow improvements can be considered as direct improvements to hardware circuit structures. Designers can mostly obtain the corresponding hardware circuit structures by programming the improved method flows into the hardware circuits. Therefore, it cannot be said that method flow improvements cannot be realized by hardware physical modules. For example, Programmable Logic Devices (PLDs) (e.g. Field Programmable Gate Arrays (FPGAs)) are such integrated circuits, whose logical functions are determined by the user's programming of the device. Instead of chip manufacturers designing and manufacturing dedicated integrated circuit chips, designers program and "integrate" digital systems onto a single PLD.Nowadays, instead of handcrafting integrated circuit chips, this programming is mostly realized using software called a "logic compiler", which is similar to a software compiler used to write a program, and in order to compile the original code, it needs to be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one type of HDL, but many types such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used at present are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. It will be clear to those skilled in the art that a hardware circuit that realizes a logical method flow can be easily obtained by simply programming a method flow logically in some of the above hardware description languages ​​and programming it into an integrated circuit.

[0131] The controller may be realized in any suitable manner, for example, the controller may take the form of a microprocessor or processor, a computer-readable storage medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers, examples of which include, but are not limited to, microcontrollers such as ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, Silicone Labs C8051F320, etc., and the memory controller may also be realized as part of the control logic of the memory. It will be clear to those skilled in the art that in addition to implementing the controller purely with computer-readable program code, it is entirely possible to logically program method steps to cause the controller to perform the same functions in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. Thus, such a controller may be considered as a hardware component, and the devices included therein for implementing various functions may also be considered as structures within the hardware component. Alternatively, the apparatus for implementing various functions may be considered to be software modules implementing a method, or structures within hardware components.

[0132] The system, device, module or unit described in the above embodiments may be specifically realized by a computer chip, an entity, or a product having some function. A typical realizing device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet, a wearable device, or any combination of these devices.

[0133] For convenience of explanation, the above-mentioned device will be described by dividing it into various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be realized by the same or multiple pieces of software and / or hardware.

[0134] As will be appreciated by those skilled in the art, the embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Thus, the present disclosure may take the form of an embodiment consisting of only hardware, an embodiment consisting of only software, or an embodiment consisting of a combination of software and hardware. Furthermore, the present disclosure may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0135] The present disclosure will be described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, may be realized by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, whereby the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0136] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0137] These computer program instructions may be loaded into a computer or other programmable data processing device such that a sequence of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0138] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0139] The memory may include volatile, random access memory (RAM) and / or non-volatile memory forms of computer readable storage media, such as read only memory (ROM) or flash memory (flash RAM). The memory is one example of a computer readable storage medium.

[0140] Computer-readable storage media include non-volatile and volatile media, movable and non-movable media, and may implement information storage by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, Phase Change Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassette tape, magnetic tape magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer readable storage media does not include transitory computer readable storage media, such as modulated data signals and carriers.

[0141] Additionally, the terms "comprise", "contain" or any other variation thereof are intended to include a non-exclusive inclusion, whereby a process, method, article or device that comprises a set of elements not only includes those elements, but also includes other elements not expressly listed, or includes the inherent elements of such process, method, article or device. Absent more limitations, an element qualified by the phrase "comprises a..." does not exclude the presence of further identical elements in a process, method, article or device that includes said element.

[0142] The disclosure may be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0143] Each embodiment in the present disclosure will be described in a stepwise manner, and the same or similar parts between each embodiment may be referred to each other, and the emphasis of each embodiment is on the differences from other embodiments. In particular, the system embodiment is basically similar to the method embodiment, so it will be briefly described, and the relevant parts may be referred to the description of part of the method embodiment.

[0144] The above is merely an embodiment of the present disclosure, and is not used to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and principle of the present disclosure should be included in the scope of the claims of the present disclosure.

Claims

1. obtaining sample data and a target model; dividing the target model into a plurality of sub-models, each of the plurality of sub-models including a portion of a network layer in the target model; determining at least one computing node for arranging the sub-models based on the sub-models, creating a plurality of containers on the at least one computing node, and arranging the sub-models in the plurality of containers, respectively; running a model training task using the sample data to train the sub-models disposed within the containers; obtaining load data of the at least one computing node when executing a model training task; determining, for each of the plurality of containers, a calculation time of the sub-model when executing a training task of the sub-model arranged in the container as a calculation time corresponding to the container; determining a computation node that requires container distribution adjustment as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers; adjusting a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other; and executing a training task of the target model based on each computing node on which the distribution of containers has been adjusted. A distributed training method for container scheduling for intelligent computing, comprising:

2. The step of dividing the target model to obtain a plurality of the sub-models includes: determining a computation time of the target model when performing a model training task; and dividing a network layer included in the target model based on a calculation time of the target model to obtain the plurality of sub-models.

2. The method of claim 1 .

3. The step of determining, for each of the plurality of containers, a computation time of the sub-model when executing a training task of the sub-model disposed in the container, comprises: determining training statistics corresponding to the container from a given shared storage system; determining a calculation time for the sub-model placed in the container when executing a training task for the sub-model based on a start time and an end time for executing a training task for the sub-model placed in the container, the start time and the end time being included in the training statistical information; The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, and the target logs are filtered from the logs generated by each of the at least one computing node according to a predetermined designated keyword, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

2. The method of claim 1 .

4. The step of determining a computation node that needs to adjust the distribution of containers based on the load data of the at least one computation node and the computation time corresponding to each of the plurality of containers includes: Sorting the plurality of containers in descending order of the operation times corresponding to each of the plurality of containers to obtain a first sorting result; determining a container that is located before a predetermined order in the first sorting result as a target container; determining a computing node that needs to adjust a distribution of containers based on the target container and load data of the at least one computing node.

2. The method of claim 1 .

5. The step of determining a computing node that needs to adjust the distribution of containers based on the target container and load data of the at least one computing node includes: determining a computing node on which the target container is placed as a first node; determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers in the first node as a second node from among other computation nodes; determining the first node and the second node as computation nodes that need to adjust a distribution of containers; 5. The method of claim 4.

6. The step of determining a computing node that needs to adjust the distribution of containers based on the target container and load data of the at least one computing node includes: determining, when it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, a calculation node on which a new container to be created is to be placed as a calculation node on which a distribution of containers needs to be adjusted based on load data of the at least one calculation node; The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes: The method includes a step of creating a new container in the target node, creating a copy of model data of a sub-model placed in the target container, and placing the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to the respective plurality of containers are close to each other.

5. The method of claim 4.

7. The step of determining a computing node on which a new container to be created is to be placed as a computing node that needs to adjust distribution of the container based on load data of the at least one computing node, sorting the computing nodes other than the computing node on which the target container is placed in order of increasing load data of the at least one computing node to obtain a second sorting result; Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result; For any two sorted adjacent computing nodes among the other computing nodes, If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node on which a new container to be created is to be placed; if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, determining whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which a new container to be created is placed is determined.

7. The method of claim 6.

8. determining, when it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a sub-model having a network layer dependency relationship with the sub-model corresponding to the new container to be created as a related sub-model; determining a computation node on which the associated sub-model is located as an associated node; measuring a network delay between the relevant node and another computing node other than the relevant node; and determining a computing node on which to place a new container to be created from among computing nodes other than the associated node based on the network delay obtained by the measurement.

8. The method of claim 7.

9. The step of determining a computing node that needs to adjust the distribution of containers based on the target container and load data of the at least one computing node includes: determining a computing node on which the target container is located; a step of determining, when it is determined that a designated container has been placed on the computation node on which the target container has been placed, that the computation node on which the target container has been placed is a computation node on which a distribution of containers needs to be adjusted, and a sub-model placed on the designated container is the same as the sub-model placed on the target container; The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes: The method includes a step of deleting the target container or the designated container from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

5. The method of claim 4.

10. The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes: and adjusting the distribution of containers in the target node, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other and that the loads of the at least one calculation node are close to each other. The method according to any one of claims 1 to 9.

11. a first acquisition module configured to acquire sample data and an object model; a segmentation module configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model; a first determination module configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively; a first training module configured to perform a model training task using the sample data to train the sub-models disposed within the containers; a second acquisition module configured to acquire load data of the at least one computing node when executing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container; A second determination module configured to determine a computation node that needs to adjust the distribution of containers as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers; an adjustment module configured to adjust a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other; and a second training module configured to execute a training task of the target model based on each computing node to which the distribution of containers has been adjusted. A distributed training apparatus for container scheduling for intelligent computing, comprising:

12. A computer-readable storage medium storing a computer program, the computer program being executed by a processor to perform the method according to any one of claims 1 to 9. A computer-readable storage medium comprising:

13. An electronic device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, the electronic device performing the method according to any one of claims 1 to 9 when the processor executes the computer program.

1. An electronic device comprising:

Images