Model training method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method and device, electronic equipment and a storage medium, and relates to the technical field of distributed computing. The method comprises the steps of dividing a context into context tasks based on structural analysis of a to-be-trained model; performing dependency relationship analysis on the context task to determine a task dependency graph; configuring priorities for the context tasks according to the task dependency graph, distributing the context tasks with the configured priorities to corresponding computing nodes, training according to an execution sequence, and obtaining load information and task execution information of the computing nodes; and according to the load information and the task execution information of the computing nodes, scheduling context tasks among the computing nodes, adjusting the execution sequence of the context tasks in the computing nodes, and outputting a training result. According to the real-time load condition of each node, task allocation is adjusted, load balancing of the computational nodes is realized, the utilization efficiency of computational resources is improved, a context task scheduling strategy is optimized, the utilization rate of the computational resources is improved, and the training time is shortened.

Description

Background Art

[0002] In the field of deep learning, as the scale of models continues to grow, the computing resources and time required to train complex and large models have increased exponentially. In order to meet this demand, distributed model training methods have gradually become mainstream. By distributing training tasks to multiple computing nodes, distributed training can effectively utilize parallel computing capabilities and significantly accelerate the model training process. However, as the complexity of the model increases, traditional distributed training methods face many challenges in task scheduling and resource management. There are problems such as low computing resource utilization due to uneven node load, limited task execution order and parallelism, which prolongs the training time.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a model training method, device, electronic device and storage medium, which at least to a certain extent overcome the problems of low computing resource utilization and long training time.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0007] Based on the structural analysis of the model to be trained, the context is divided into context tasks;

[0008] Performing dependency analysis on the context tasks to determine a task dependency graph;

[0009] According to the task dependency graph, the context task is configured with a priority, and the context task with the configured priority is assigned to the corresponding computing node for training in an execution order, and the load information and task execution information of the computing node are obtained;

[0010] According to the load information and task execution information of the computing nodes, the context tasks are scheduled between the computing nodes, the execution order of the context tasks within the computing nodes is adjusted, and the training results are output.

[0011] In some embodiments, based on the structural analysis of the model to be trained, the context is divided into context tasks, including:

[0012] Performing structural analysis on the model to be trained to determine a context list; the context list includes multiple contexts;

[0013] calculating the complexity of the plurality of contexts and dependencies between the plurality of contexts;

[0014] Each of the multiple contexts is divided into context tasks according to complexity of the multiple contexts and dependencies between the multiple contexts.

[0015] In some embodiments, a structural analysis is performed on the model to be trained to determine a context list, including:

[0016] Performing structural analysis on the topological structure of the model to be trained, identifying the components and modules of the model to be trained, and determining the architecture of the model to be trained;

[0017] According to the architecture of the model to be trained, multiple contexts that are related in sequence are identified to form a context list.

[0018] In some embodiments, performing dependency analysis on the context tasks to determine a task dependency graph includes:

[0019] Determining the training algorithm properties of the context task; the training algorithm properties of the context task include: forward calculation, back propagation, and gradient update;

[0020] Classifying the context tasks according to the training algorithm properties of the context tasks to determine task nodes; wherein the task nodes have location information;

[0021] Analyze the data dependency, calculation order dependency and resource dependency between the task nodes to determine the dependency edge;

[0022] A directed acyclic graph is drawn according to the dependency edges and the task nodes and their position information to determine a task dependency graph.

[0023] In some embodiments, configuring a priority for the context task according to the task dependency graph, and assigning the context task with the configured priority to a corresponding computing node for training in an execution order, and obtaining load information and task execution information of the computing node, including:

[0024] configuring a priority for the context task according to the task dependency graph;

[0025] Determine the execution order of the context tasks with configured priorities according to different priorities, and organize the context tasks into a task queue according to the execution order;

[0026] Configuring a load balancing strategy based on the performance information of the computing node;

[0027] Allocating the task queue to a corresponding computing node according to the load balancing strategy, so that the computing node trains the context tasks in the task queue according to the execution order;

[0028] Obtain load information and task execution information of the computing node during the training process.

[0029] In some embodiments, according to the load information and task execution information of the computing nodes, scheduling the context tasks between the computing nodes and adjusting the execution order of the context tasks within the computing nodes, and outputting the training results include:

[0030] Configure a task scheduling strategy according to the priority of the context task and the execution order of the task queue;

[0031] According to the load information of the computing node, if it is detected that the load of the computing node exceeds a set threshold, determining the context task to be migrated and the target computing node according to the load balancing strategy;

[0032] Determining an embedding position of a task queue in the target computing node according to the task scheduling strategy;

[0033] Migrating the context task to be migrated to the embedded position of the task queue in the target computing node, and determining an updated task queue and an updated execution order;

[0034] The target computing node continues to perform training on the updated task queue according to the updated execution order;

[0035] If all computing nodes complete training, the training results are output.

[0036] In some embodiments, it also includes:

[0037] According to the task execution information, if it is detected that there is a context task to be scheduled in the computing node, the priority of the context task to be scheduled is adjusted according to the task scheduling strategy to modify the execution order of the current task queue in which the context task to be scheduled is located; the context task to be scheduled is a context task whose actual execution time exceeds the expected execution time by a set monitoring threshold;

[0038] Determine, according to the modified execution order and task dependency graph of the current task queue, an associated task queue having a dependency relationship with the current task queue; the associated task queue and the current task queue are located in the same computing node or in different computing nodes;

[0039] The execution order of the associated task queues is adjusted according to the task scheduling strategy and the task dependency graph.

[0040] In some embodiments, it also includes:

[0041] According to the task execution information, if it is detected that there is a context task to be scheduled in the computing node, a target scheduling computing node is determined according to a task scheduling strategy and a load balancing strategy;

[0042] Obtaining the execution order of the task queue in the target scheduling computing node;

[0043] Determine the embedded position to be scheduled according to the execution order of the task queue in the target scheduling computing node and the task dependency graph;

[0044] The priority of the target scheduling computing node is modified and scheduled to the embedded position to be scheduled, and the scheduled task queue and the scheduled execution order in the target scheduling computing node are determined.

[0045] According to another aspect of the present disclosure, a model training device is also provided, comprising:

[0046] A context division module, which is used to divide the context into context tasks based on the structural analysis of the model to be trained;

[0047] A task dependency graph determination module, used to perform dependency analysis on the context tasks and determine a task dependency graph;

[0048] A context task allocation module is used to configure priorities for the context tasks according to the task dependency graph, and allocate the context tasks with configured priorities to corresponding computing nodes for training in an execution order, and obtain load information and task execution information of the computing nodes;

[0049] The task scheduling and execution order adjustment module is used to schedule the context tasks between the computing nodes and adjust the execution order of the context tasks within the computing nodes according to the load information and task execution information of the computing nodes, and output the training results.

[0050] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned model training methods by executing the executable instructions.

[0051] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model training method described in any one of the above is implemented.

[0052] According to another aspect of the present disclosure, a computer program product is also provided, including: a computer program or instructions, which implement any of the above-mentioned model training methods when executed by a processor.

[0053] The model training method, device, electronic device and storage medium provided in the embodiments of the present disclosure include: dividing the context into context tasks based on the structural analysis of the model to be trained; performing dependency analysis on the context tasks to determine a task dependency graph; configuring priorities for the context tasks according to the task dependency graph, and assigning the context tasks with configured priorities to corresponding computing nodes for training in an execution order, and obtaining load information and task execution information of the computing nodes; scheduling the context tasks between the computing nodes and adjusting the execution order of the context tasks within the computing nodes according to the load information and task execution information of the computing nodes, and outputting the training results. The present disclosure can adaptively divide the context into context tasks according to the structure of the model and the characteristics of the training data by performing structural analysis on the training model, obtain a task dependency graph by performing accurate dependency analysis on the context tasks, and the task dependency graph can clearly show the dependency relationship between each context task, thereby providing a basic basis for task scheduling optimization, configuring priorities for context tasks and assigning them to computing nodes for training in an execution order, optimizing the execution order and parallelism of context tasks based on the dependency relationship, making full use of computing resources, improving the overall training performance, and giving priority to scheduling tasks with less dependency; by obtaining the load information and task information of computing nodes during the training process, The task execution information is used to schedule context tasks between computing nodes and adjust the execution order of context tasks within computing nodes. During the training process, task allocation is dynamically adjusted according to the real-time load of each node, achieving load balancing of each computing node, solving the problems of uneven node load and low resource utilization that are common in traditional distributed training methods, avoiding the problem of node overload or idleness, thereby improving the utilization efficiency of computing resources. At the same time, the scheduling strategy of context tasks is optimized, effectively improving the utilization rate of computing resources, shortening the training time, and significantly improving the overall efficiency of distributed training through effective computing task allocation. Based on the task dependency graph, the parallelism of different tasks can be evaluated. By analyzing the dependencies between tasks, it is determined which tasks can be executed simultaneously and which tasks must be executed after the completion of the previous tasks, which improves the parallel execution efficiency of tasks, reduces unnecessary waiting time, and achieves the optimal utilization of computing resources. By introducing adaptive context partitioning and optimized task scheduling strategies, the efficiency of large-scale distributed model training is significantly improved, overcoming the shortcomings of existing technologies in load balancing and dependency management. This method is particularly suitable for large model training tasks in complex dependencies and dynamic load environments, and has broad application prospects in fields such as artificial intelligence, computer vision, and recommendation systems.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0056] Figure 1 A schematic diagram showing the system structure of a model training method in an embodiment of the present disclosure.

[0057] Figure 2 A schematic diagram of a model training method in an embodiment of the present disclosure is shown.

[0058] Figure 3 A schematic diagram of the process of dividing a context into context tasks in a model training method in an embodiment of the present disclosure is shown.

[0059] Figure 4 A schematic diagram of a process of determining a context list in a model training method in an embodiment of the present disclosure is shown.

[0060] Figure 5 A schematic diagram of the process of determining a task dependency graph of a model training method in an embodiment of the present disclosure is shown.

[0061] Figure 6 A schematic diagram of a context task allocation process of a model training method in an embodiment of the present disclosure is shown.

[0062] Figure 7 A schematic diagram of the context task scheduling and execution order adjustment process of a model training method in an embodiment of the present disclosure is shown.

[0063] Figure 8 A flow chart of a model training method in an embodiment of the present disclosure is shown.

[0064] Fig. 9 A schematic diagram of a model training device in an embodiment of the present disclosure is shown.

[0065] Fig.10 A schematic diagram of a computer-readable storage medium showing a model training method in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0067] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0068] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0069] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the model training method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0070] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0071] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0072] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0073] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0074] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0075] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0076] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0077] Under the above system architecture, a model training method is provided in an embodiment of the present disclosure, which can be executed by any electronic device with computing and processing capabilities.

[0078] In some embodiments, the model training method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the model training method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the model training method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0079] The main problems of existing distributed model training methods include uneven node load, strong task dependency, and low computing resource utilization. When allocating tasks, existing distributed training methods often fail to fully consider the load conditions of different computing nodes, resulting in some nodes being overloaded while other nodes are idle. This uneven load situation will greatly reduce the overall computing efficiency of the system. In the training process of large models, there are often complex dependencies between tasks, and the tasks are highly dependent. Due to the lack of effective management of these dependencies, the execution order and parallelism of tasks are limited, resulting in low resource utilization during training. Due to uneven node load and poor task dependency management, a large number of computing resources in the system are idle and fail to give full play to their performance advantages. This not only prolongs the training time, but also wastes precious computing resources.

[0080] Figure 2 A schematic diagram of a model training method in an embodiment of the present disclosure is shown. Figure 2 As shown, the model training method provided in the embodiment of the present disclosure includes the following steps:

[0081] Step S202: Divide the context into context tasks based on the structural analysis of the model to be trained;

[0082] Step S204: Analyze the dependency relationship of the context tasks to determine a task dependency graph;

[0083] Step S206: configuring priorities for context tasks according to the task dependency graph, and assigning the context tasks with configured priorities to corresponding computing nodes for training in an execution order, and obtaining load information and task execution information of the computing nodes;

[0084] Step S208: According to the load information and task execution information of the computing nodes, the context tasks are scheduled between the computing nodes and the execution order of the context tasks within the computing nodes is adjusted, and the training results are output.

[0085] The present disclosure can adaptively divide the context into context tasks according to the structure of the model and the characteristics of the training data by performing structural analysis on the training model, obtain a task dependency graph by performing accurate dependency analysis on the context tasks, and the task dependency graph can clearly show the dependency relationship between each context task, thereby providing a basic basis for task scheduling optimization, configuring priorities for context tasks and assigning them to computing nodes for training in an execution order, optimizing the execution order and parallelism of context tasks based on the dependency relationship, making full use of computing resources, improving the overall training performance, and giving priority to scheduling tasks with less dependency; by obtaining the load information and task information of computing nodes during the training process, The task execution information is used to schedule context tasks between computing nodes and adjust the execution order of context tasks within computing nodes. During the training process, task allocation is dynamically adjusted according to the real-time load of each node, achieving load balancing of each computing node, solving the problems of uneven node load and low resource utilization that are common in traditional distributed training methods, avoiding the problem of node overload or idleness, thereby improving the utilization efficiency of computing resources. At the same time, the scheduling strategy of context tasks is optimized, effectively improving the utilization rate of computing resources, shortening the training time, and significantly improving the overall efficiency of distributed training through effective computing task allocation. Based on the task dependency graph, the parallelism of different tasks can be evaluated. By analyzing the dependencies between tasks, it is determined which tasks can be executed simultaneously and which tasks must be executed after the completion of the previous tasks, which improves the parallel execution efficiency of tasks, reduces unnecessary waiting time, and achieves the optimal utilization of computing resources. By introducing adaptive context partitioning and optimized task scheduling strategies, the efficiency of large-scale distributed model training is significantly improved, overcoming the shortcomings of existing technologies in load balancing and dependency management. This method is particularly suitable for large model training tasks in complex dependencies and dynamic load environments, and has broad application prospects in fields such as artificial intelligence, computer vision, and recommendation systems.

[0086] Figure 3A schematic diagram of a process of dividing a context into context tasks in a model training method according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, in the embodiment, based on the structural analysis of the model to be trained, the context is divided into context tasks, including:

[0087] Step S302: Perform structural analysis on the model to be trained to determine a context list; the context list includes multiple contexts;

[0088] Step S304: Calculate the complexity of multiple contexts and the dependencies between the multiple contexts;

[0089] Step S306: Divide each of the multiple contexts into context tasks according to the complexity of the multiple contexts and the dependencies between the multiple contexts.

[0090] The model to be trained usually has a load structure, which includes different layers of neural networks, specific sub-networks, or complex computing units. Through structural analysis, multiple contexts can be obtained, and the context list is composed of multiple contexts. According to the characteristics and requirements of each context, the complexity of the context is calculated. The complexity mainly includes multiple dimensions such as computing cost, memory requirements, and data dependencies. The calculation identifies the dependencies between multiple contexts and ensures that these dependencies are taken into account when assigning tasks to avoid conflicts or bottlenecks during training. According to the complexity and dependencies, each context is divided into multiple context tasks to optimize training efficiency and resource utilization.

[0091] Figure 4 A schematic diagram of a process of determining a context list in a model training method according to an embodiment of the present disclosure is shown. Figure 4 As shown, in the embodiment, a structural analysis is performed on the model to be trained to determine a context list, including:

[0092] Step S402: Perform structural analysis on the topological structure of the model to be trained, identify the components and modules of the model to be trained, and determine the architecture of the model to be trained;

[0093] Step S404: According to the architecture of the model to be trained, identify multiple contexts that are related in sequence to form a context list.

[0094] When performing structural analysis on the model to be trained, first, the topological structure of the model to be trained is analyzed in detail to identify the various components, computing modules or functional modules in the model. These modules can be different layers of a neural network, specific subnetworks, or complex computing units; the structure of the model is analyzed to identify its main components and modules. This process helps to understand the overall architecture of the model and determine which parts can be processed independently. By analyzing the computational complexity, data dependencies and possibility of parallel execution of each module in the model to be trained, a suitable context partitioning strategy is determined. According to the architecture of the model to be trained, the model to be trained is divided using the context partitioning strategy and multiple contexts related in sequence are identified to form a context list. The identified contexts can represent a group of computing tasks with similar computing characteristics or interdependence. By identifying the computational complexity and resource requirements of each context, the division method of different contexts in the distributed system is determined. Therefore, the context is divided into multiple context tasks based on the complexity and dependency of the context. In essence, a group of computing tasks are divided according to different requirements when applied, thereby realizing the adaptive division of context tasks. Different contexts are identified according to the model structure. These contexts can be understood as different parts or tasks of the model, and each context may require different resources or computing power to train.

[0095] Figure 5 A schematic diagram of the process of determining a task dependency graph of a model training method in an embodiment of the present disclosure is shown. Figure 5 As shown, in the embodiment, performing dependency analysis on context tasks to determine a task dependency graph includes:

[0096] Step S502: determining the training algorithm properties of the context task; the training algorithm properties of the context task include: forward calculation, back propagation, and gradient update;

[0097] Step S504: classifying the context tasks according to the training algorithm properties of the context tasks, and determining the task nodes; wherein the task nodes have position information;

[0098] Step S506: Analyze the data dependency, calculation order dependency and resource dependency between task nodes, and determine the dependency edge;

[0099] Step S508: Draw a directed acyclic graph based on the dependency edges and task nodes and their position information to determine the task dependency graph.

[0100] In large-scale distributed model training, the dependencies between different contexts may be very complex. These dependencies directly affect the parallelism and execution order of tasks. If handled improperly, some nodes in the system may be in a waiting state for a long time, affecting the overall training efficiency. This embodiment proposes a context-dependent task scheduling strategy, which optimizes the task scheduling order and parallelism, makes full use of computing resources, and improves the overall training performance. To implement this task scheduling strategy, it is necessary to analyze the dependencies of context tasks and construct a task dependency graph.

[0101] By comprehensively analyzing the dependencies of each context task in the model training task, a task dependency graph is constructed. The task dependency graph can clearly represent the dependencies between each context task, thus providing a basic basis for the optimization of the scheduling strategy.

[0102] According to multiple context tasks, the training algorithm properties of context tasks are determined, which are mainly divided into forward calculation, back propagation and gradient update, etc.; according to the training algorithm properties of context tasks, context tasks are classified to obtain task nodes. The above classification process also determines the position information of each task node in the task dependency graph. The data dependency, calculation order dependency and resource dependency between task nodes are analyzed to obtain the task dependency relationship, thereby determining the dependency edge; based on the obtained node, i.e. the node position, a complete directed acyclic graph is drawn in combination with the dependency edge to represent the task dependency graph.

[0103] Figure 6 A schematic diagram of a context task allocation process of a model training method in an embodiment of the present disclosure is shown. Figure 6 As shown, in the embodiment, according to the task dependency graph, the context task is configured with a priority, and the context task with the configured priority is assigned to the corresponding computing node for training in the execution order, and the load information and task execution information of the computing node are obtained, including:

[0104] Step S602: configuring a priority for the context task according to the task dependency graph;

[0105] Step S604: determining the execution order of the context tasks with configured priorities according to different priorities, and organizing the context tasks into a task queue according to the execution order;

[0106] Step S606: configuring a load balancing strategy based on the performance information of the computing nodes;

[0107] Step S608: Allocate the task queue to the corresponding computing node according to the load balancing strategy, so that the computing node trains the context tasks in the task queue according to the execution order;

[0108] Step S6010: Obtain the load information and task execution information of the computing node during the training process.

[0109] According to the task dependency graph obtained by analyzing the dependency relationship of context tasks, the priority of context tasks is configured; the priority is divided into high priority, medium priority and low priority according to the level, among which high priority is executed first and low priority is executed last. In one case, high priority refers to context tasks with less or no dependency, so high-priority context tasks will be scheduled for training execution first to increase the number of parallel computing tasks, thereby reducing the waiting time of nodes in the system and improving the overall computing efficiency. Context tasks are sorted according to priority to determine the execution order, and context tasks of different priorities are organized into task queues according to the execution order to ensure that high-priority context tasks can be executed first. Context tasks with fewer dependencies are scheduled in parallel as much as possible to improve the computing resource utilization of the system.

[0110] A monitoring system is established for all computing nodes to monitor the load information of computing nodes to ensure that the system remains balanced during the training process and make timely adjustments. The load balancing strategy is configured based on the performance information of the computing nodes. The load balancing strategy is responsible for allocating different context tasks to appropriate computing nodes during the training process. The performance information of the computing nodes mainly includes CPU main frequency, memory utilization, GPU utilization, hard disk utilization, and I / O interface rate. The load balancing strategy can use a load balancer in this embodiment.

[0111] According to the load balancing strategy, the task queue is assigned to the most appropriate computing node. This step is an important step to ensure the workload of each node is balanced. The computing node receives the task queue and trains the context tasks in the task queue according to the execution order; at this point, each computing node starts to execute the training tasks assigned to them, that is, the task queue containing the context tasks. This is the core part of the training process. Multiple computing nodes work in parallel to speed up the overall training progress.

[0112] The monitoring system continuously monitors and obtains the load information and task execution information of computing nodes during the training process, ensuring that performance degradation or training failure will not occur due to overloading of some nodes during the training process.

[0113] Figure 7 A schematic diagram of the context task scheduling and execution order adjustment process of a model training method in an embodiment of the present disclosure is shown, Figure 7 As shown, in the embodiment, according to the load information and task execution information of the computing nodes, context tasks are scheduled between computing nodes and the execution order of context tasks within the computing nodes is adjusted, and training results are output, including:

[0114] Step S702: configuring a task scheduling strategy according to the priority of the context task and the execution order of the task queue;

[0115] Step S704: according to the load information of the computing node, if the load of the computing node is detected to exceed the set threshold, the context task to be migrated and the target computing node are determined according to the load balancing strategy;

[0116] Step S706: Determine the embedding position of the task queue in the target computing node according to the task scheduling strategy;

[0117] Step S708: Migrate the context task to be migrated to the embedded position of the task queue in the target computing node, and determine the updated task queue and the updated execution order;

[0118] Step S7010: the target computing node continues to perform training on the updated task queue according to the updated execution order;

[0119] Step S7012: If all computing nodes have completed training, output the training results.

[0120] Based on configuring priorities for context tasks and sorting them to obtain the execution order of the task queue, a task scheduling strategy is configured to achieve dynamic scheduling of tasks. This task scheduling strategy will dynamically schedule context tasks during the training process, and the task scheduling strategy will also work with the load balancing strategy to maximize the overall efficiency of the training process. The monitoring system detects the load information of all computing nodes to determine whether the current load is balanced. If the load of the measured computing node exceeds the set threshold, it means that the current computing node is overloaded, and some context tasks in the current computing node need to be migrated to achieve load balancing. Following the load balancing strategy, the context tasks that cause excessive load in the current computing node are identified as context tasks to be migrated, and new, lightly loaded computing nodes are found in other computing nodes as target computing nodes for the context tasks to be migrated. Since there is an executing task queue in the target computing node, if you want to add the context task to be migrated to the task queue, you cannot simply put it directly into the target computing node, which will disrupt the execution of the target computing node. Therefore, it is necessary to determine the embedding position of the task queue in the target computing node according to the task scheduling strategy; the embedding position does not affect the training plan of the task queue in the target computing node, so the context task to be migrated is migrated to the embedding position of the task queue in the target computing node, and the task queue and the execution order are updated, and the updated task queue and the updated execution order are determined; the target computing node continues to perform training on the updated task queue according to the updated execution order to rebalance the system load, and after completing the necessary adjustments, continue the distributed training task.

[0121] During the training process, the feedback data generated during the training process is collected and analyzed to better understand whether the current context allocation strategy is effective, establish a real-time feedback mechanism, and adjust the load balancing strategy based on the collected feedback to further optimize the context allocation and system performance. In an embodiment, the context division method is adaptively adjusted according to the load information of the computing node during the training process and the load balancing process of migrating the context tasks to be migrated to the target computing node. The execution of the training tasks is continuously monitored, and the context task allocation on each computing node is dynamically adjusted through the feedback mechanism to achieve load balancing. Check the overall progress of the training. If the training progress reaches 100%, the training end stage is entered and the training results are output. These training results may include performance indicators, training time, resource utilization, etc. of the model, providing a basis for subsequent model application or further optimization.

[0122] During the training process, the context partitioning strategy is dynamically adjusted by real-time monitoring of the CPU, GPU, memory and other resource usage of each node. Context tasks are adaptively allocated based on load information to avoid overload or idle resources on some nodes. To further ensure load balance, context tasks are dynamically migrated during the training process. For example, when a node is overloaded, some context tasks can be migrated to computing nodes with lighter loads, thereby achieving overall balanced distribution of computing resources. A real-time feedback mechanism is established to continuously optimize the context partitioning strategy by monitoring the training progress and computing node load to ensure balanced computing load on each node during training. This feedback mechanism can respond to load changes during training in a short time, thereby improving the robustness of the system and training efficiency.

[0123] In an embodiment, the model training method also includes: based on the task execution information, if it is detected that there is a context task to be scheduled in the computing node, then according to the task scheduling strategy, the priority of the context task to be scheduled is adjusted to modify the execution order of the current task queue where the context task to be scheduled is located; the context task to be scheduled is a context task whose actual execution time exceeds the expected execution time by a set monitoring threshold; based on the modified execution order of the current task queue and the task dependency graph, an associated task queue that has a dependency relationship with the current task queue is determined; the associated task queue and the current task queue are in the same computing node or in different computing nodes; according to the task scheduling strategy and the task dependency graph, the execution order of the associated task queue is adjusted.

[0124] Based on the task dependency graph, the degree of parallelism of different context tasks can be evaluated. By analyzing the dependencies between context tasks, it is determined which context tasks can be executed simultaneously and which context tasks must be executed after the completion of the previous context tasks. This is used to schedule context tasks during training.

[0125] During the training process, the task scheduling strategy is dynamically adjusted according to the actual execution status and feedback information of task completion. For example, if it is found that the actual execution time of some context tasks is longer than expected, the scheduling order of subsequent context tasks can be appropriately adjusted to reduce idle time and resource waste in the system. Specifically, the context tasks to be scheduled can be scheduled within the same computing node or across computing nodes. The context tasks whose actual execution time exceeds the expected execution time by more than the set monitoring threshold are regarded as the context tasks to be scheduled. According to the task scheduling strategy, the priority of the context tasks to be scheduled is first adjusted. Generally, the priority of the context tasks is lowered, and other context tasks are executed first, and the execution of the context tasks to be scheduled is postponed to reduce idle time and resource waste in the system. On the basis of adjusting the priority to modify the execution order of the task queue, due to the dependency between context tasks, the execution order of the context tasks to be scheduled in the current task queue is adjusted. Then the associated task queues that have a dependency relationship with the current task queue also need to be adjusted accordingly. In one case, the current task queue and the associated task queue are in the same computing node. In another case, the current task queue and the associated task queue are not in the same computing node, that is, they are in two computing nodes respectively. Both cases require task scheduling strategies and task dependency graphs to adjust the execution order of the associated task queues. Based on the modified execution order of the current task queue, continue to execute training. Based on the adjusted execution order of the associated task queue, continue to execute training.

[0126] In an embodiment, the model training method also includes: based on the task execution information, if it is detected that there are context tasks to be scheduled in the computing node, then according to the task scheduling strategy and the load balancing strategy, determining the target scheduling computing node; obtaining the execution order of the task queue in the target scheduling computing node; determining the embedded position to be scheduled according to the execution order of the task queue in the target scheduling computing node and the task dependency graph; modifying the priority of the target scheduling computing node and scheduling it to the embedded position to be scheduled, and determining the scheduled task queue and the scheduled execution order in the target scheduling computing node.

[0127] During the training process, in one case, the context task to be scheduled needs to be directly scheduled to other computing nodes. If it is detected that the actual execution time of the context task to be scheduled is longer than the expected execution time by more than the set monitoring threshold, the target scheduling computing node is determined according to the task scheduling strategy and the load balancing strategy; since there is already a task queue that is executing training in the target scheduling computing node, the execution order of the task queue in the target scheduling computing node is obtained; according to the execution order of the task queue in the target scheduling computing node and the task dependency graph, the embedded position to be scheduled is determined; similarly, the embedded position to be scheduled does not affect the training plan of the task queue in the target scheduling computing node, the priority of the target scheduling computing node is modified and scheduled to the embedded position to be scheduled, the scheduled task queue and the scheduled execution order in the target scheduling computing node are determined, and the target scheduling computing node executes the scheduled task queue according to the scheduled execution order and continues the model training.

[0128] Adjustments refer to adjustments on the same computing node or globally. For example, if a task on a node takes too long to execute, the system can adjust the execution order of other tasks to avoid idle resources. This can involve reordering tasks on the same node, or it may include cross-node task scheduling optimization, with the goal of improving the throughput and efficiency of the entire system.

[0129] During the training process, we not only focus on the order of task scheduling, but also optimize scheduling in combination with load monitoring information to ensure that the task scheduling strategy is coordinated with the load balancing strategy to maximize the overall efficiency of the training process. By rationally analyzing the dependencies between contexts and optimizing the order of task scheduling, we significantly improve the parallelism of tasks, thereby reducing unnecessary waiting time. The optimization of task scheduling strategies enables more efficient use of computing resources, reduces idle resources and waste, and improves the overall training speed.

[0130] In the disclosed embodiments, by dynamically adjusting task division and optimizing scheduling strategies, the efficiency of large-scale distributed model training is significantly improved, training time is shortened, computing resources are saved, and training efficiency is improved. More efficient resource utilization reduces the waste of computing resources, reduces training costs, and makes large-scale model training more economical. It will promote the popularization of ultra-large-scale artificial intelligence models in more fields, make more complex and efficient AI systems possible, promote the popularization of large-scale AI applications, and thus promote the development of the entire industry.

[0131] The disclosed embodiments also provide a model training method, which is applied to the following scenarios: Large-scale enterprise-level AI model training: In enterprises that need to train complex and large-scale deep learning models, such as finance, medical care, autonomous driving and other fields, the method of the present invention can greatly improve the training efficiency and shorten the model development cycle. Scientific computing and simulation: In scientific computing fields such as astrophysics, materials science, and bioinformatics, the training and simulation of ultra-large-scale models require a large amount of computing resources. The method proposed in this patent can optimize the scheduling and resource management of these compute-intensive tasks in a distributed computing environment, improve computing efficiency, and support more complex scientific research and simulation tasks. It is mainly divided into two parts: a dynamic context partitioning algorithm and a context-dependent task scheduling strategy.

[0132] The dynamic context partitioning algorithm can adaptively perform context partitioning according to the structure of the model and the characteristics of the training data, and dynamically adjust the task allocation according to the real-time load of each node during the training process. In this way, the load balancing of each computing node is achieved, avoiding the problem of node overload or idleness, thereby improving the utilization efficiency of computing resources. The context-dependent task scheduling strategy can accurately analyze the context dependencies in large model training, and optimize the execution order and parallelism of tasks based on the dependencies. By giving priority to scheduling tasks with less dependencies and dynamically optimizing the scheduling strategy, this method effectively improves the utilization of computing resources and shortens the training time.

[0133] The dynamic context partitioning algorithm can adaptively perform context partitioning based on the structural characteristics of large models and the dynamic changes of training data to achieve dynamic balance of computing load in distributed systems. This algorithm aims to solve the problems of uneven node load and low resource utilization that are common in traditional distributed training methods, and significantly improve the overall efficiency of distributed training through effective computing task allocation. Figure 8 A flow chart of a model training method in an embodiment of the present disclosure is shown. Figure 8 As shown, the main process includes:

[0134] Initialization phase: The initialization phase focuses on the basic configuration and preparation of the system to ensure that training can start smoothly.

[0135] Analyze the model structure: Analyze the structure of the model and identify its main components and modules. This process helps us understand the overall architecture of the model and determine which parts can be processed independently.

[0136] Identify context list: Based on the model structure, identify different contexts. These contexts can be understood as different parts or tasks of the model, and each context may require different resources or computing power to train.

[0137] Initialize the load balancer: Create and configure a load balancer, which is responsible for distributing different context tasks to appropriate computing nodes during the training process.

[0138] Initialize monitoring system: Set up a monitoring system that will continuously monitor the load of each node to ensure that the system remains balanced during training and make timely adjustments.

[0139] Main loop: The main loop is the core part of the training process, which continuously performs complexity calculations, context division, load balancing, and monitoring adjustments to ensure that the entire training process is efficient and balanced.

[0140] Calculate context complexity: Calculate the complexity of each context based on its characteristics and requirements. Complexity may include multiple dimensions such as computational cost, memory requirements, data dependencies, etc.

[0141] Analyze context dependencies: Identify dependencies between contexts and ensure that these dependencies are taken into account when assigning tasks to avoid conflicts or bottlenecks during training.

[0142] Dynamic context partitioning: Context tasks are partitioned and prepared for distribution to different nodes based on complexity and dependencies. This step aims to optimize training efficiency and resource utilization.

[0143] Distribute context to different nodes: Use a load balancer to distribute the divided context tasks to the most suitable computing nodes. This step is an important step to ensure that the workload of each node is balanced.

[0144] Start distributed training: Each node starts to execute the training tasks assigned to them. This is the core part of the training process, and multiple nodes work in parallel to speed up the overall training progress.

[0145] Monitor node load: The monitoring system continuously checks the load of each node to ensure that the system does not suffer performance degradation or training failure due to overload of some nodes during operation.

[0146] Load imbalance: Determine whether the current load is balanced. If the monitoring system detects that some nodes are overloaded, adjustments need to be made.

[0147] Identify overloaded contexts: If the load is unbalanced, identify those context tasks that are causing the nodes to be overloaded and find new, less loaded nodes for them.

[0148] Migrate contexts to lightly loaded nodes: Migrate identified overloaded contexts to less loaded nodes to rebalance the system load.

[0149] Continue training: After completing the necessary adjustments, continue the distributed training task.

[0150] Real-time feedback mechanism: Collect and analyze feedback data generated during the training process to better understand whether the current context allocation strategy is effective.

[0151] Update load balancing strategy: Based on the collected feedback, adjust the load balancer's strategy to further optimize context distribution and system performance.

[0152] Training completed: Check the overall progress of the training. If the training progress reaches 100%, it enters the end of training; otherwise, it returns to the main loop to continue training.

[0153] Training end: When the training is completed, the training results are output. These results may include model performance indicators, training time, resource utilization, etc., providing a basis for subsequent model application or further optimization.

[0154] In large-scale distributed model training, the dependencies between different contexts may be very complex. These dependencies directly affect the parallelism and execution order of tasks. If not handled properly, some nodes in the system may be in a waiting state for a long time, affecting the overall training efficiency. To this end, this patent proposes a context-dependent task scheduling strategy that optimizes the task scheduling order and parallelism, fully utilizes computing resources, and improves overall training performance.

[0155] Context-dependent task scheduling strategies, including:

[0156] Dependency analysis: Task dependency graph construction, by comprehensively analyzing the dependencies of each context in the model training task, a task dependency graph is constructed. This graph can clearly represent the dependencies between tasks, thus providing a basis for optimizing the scheduling strategy. Task parallelism evaluation, based on the task dependency graph, the system can evaluate the parallelism of different tasks. By analyzing the dependencies between tasks, it is determined which tasks can be executed simultaneously and which tasks must be executed after the completion of the previous task.

[0157] Steps to build a task dependency graph: Identify task nodes: Identify the various computing tasks in model training and classify them into nodes such as forward computing, back propagation, and gradient update. Determine task dependencies: Analyze the data dependencies, computing order dependencies, and resource dependencies between tasks. Build a task dependency graph: Draw task nodes and their dependency edges in the graph to form a complete directed acyclic graph to represent task dependencies.

[0158] Priority Scheduling: Priority allocation: Assign a priority to each task based on the results of task dependency analysis. Prioritize tasks with less or no dependencies to increase the number of tasks for parallel computing, thereby reducing the waiting time of nodes in the system and improving overall computing efficiency. Task queue optimization: Organize tasks of different priorities into optimized task queues to ensure that high-priority tasks are executed first. At the same time, try to schedule tasks with fewer dependencies in parallel to improve the computing resource utilization of the system.

[0159] Dynamic scheduling optimization: Dynamic adjustment of scheduling strategy: During the training process, the task scheduling strategy is dynamically adjusted according to the actual execution situation and feedback information of task completion. For example, if it is found that the actual execution time of some tasks is longer than expected, the scheduling order of subsequent tasks can be appropriately adjusted to reduce idle time and resource waste in the system. Load and scheduling collaborative optimization: The system not only pays attention to the order of task scheduling, but also combines load monitoring information for scheduling optimization to ensure that the task scheduling strategy is coordinated with the load balancing strategy to maximize the overall efficiency of the training process.

[0160] By rationally analyzing the dependencies between contexts and optimizing the order of task scheduling, the parallelism of tasks is significantly improved, thereby reducing unnecessary waiting time. The optimization of task scheduling strategies enables more efficient use of computing resources, reduces idle resources and waste, and improves the overall training speed.

[0161] Dynamic load balancing and context division, compared with the fixed task division method in the prior art, the dynamic context division algorithm proposed in this patent can adaptively adjust the distribution method of computing tasks according to the real-time load of each node during the training process. This mechanism effectively solves the problem of computing resource waste caused by uneven load in traditional distributed training, ensures the load balance of each computing node, and significantly improves the overall training efficiency.

[0162] Optimized task dependency management and scheduling strategy. Existing technologies usually have difficulty in effectively handling complex task dependencies in task scheduling, resulting in low resource utilization during training. This patent uses a context-dependent task scheduling strategy to accurately analyze the dependencies between tasks and optimize the execution order and parallelism of tasks. This strategy not only improves the parallel execution efficiency of tasks, but also reduces unnecessary waiting time and achieves optimal utilization of computing resources.

[0163] By introducing a dynamic context partitioning algorithm and an optimized task scheduling strategy, the efficiency of large-scale distributed model training is significantly improved, overcoming the shortcomings of existing technologies in load balancing and dependency management. This method is particularly suitable for large model training tasks in complex dependencies and dynamic load environments, and has broad application prospects in fields such as artificial intelligence, computer vision, and recommendation systems.

[0164] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0165] Based on the same inventive concept, the present disclosure also provides a model training device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0166] Fig. 9 A schematic diagram of a model training device in an embodiment of the present disclosure is shown. Fig. 9 As shown, the device comprises:

[0167] A context division module 901 is used to divide the context into context tasks based on the structural analysis of the model to be trained;

[0168] The task dependency graph determination module 902 is used to perform dependency analysis on the context tasks and determine the task dependency graph;

[0169] The context task allocation module 903 is used to configure the priority of the context task according to the task dependency graph, and allocate the context task with the configured priority to the corresponding computing node for training in the execution order, and obtain the load information and task execution information of the computing node;

[0170] The task scheduling and execution order adjustment module 904 is used to schedule context tasks between computing nodes and adjust the execution order of context tasks within computing nodes according to the load information and task execution information of the computing nodes, and output the training results.

[0171] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0172] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0173] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above model training methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0174] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present disclosure is described. Fig.10 The electronic device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0175] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, and a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010).

[0176] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 1010, so that the processing unit 1010 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1010 can execute the following steps of the above method embodiment: based on the structural analysis of the model to be trained, the context is divided into context tasks; the dependency analysis of the context tasks is performed to determine the task dependency graph; according to the task dependency graph, the priority is configured for the context task, and the context task with the configured priority is assigned to the corresponding computing node for training in the execution order, and the load information and task execution information of the computing node are obtained; according to the load information and task execution information of the computing node, the context task is scheduled between the computing nodes and the execution order of the context task in the computing node is adjusted, and the training result is output.

[0177] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0178] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0179] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0180] The electronic device 1000 may also communicate with one or more external devices 1040 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0181] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0182] Based on the same inventive concept, a computer-readable storage medium is also provided in the embodiment of the present disclosure, on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned model training methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above-mentioned method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0183] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0185] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0186] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0187] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, which implements the model training method of any one of the above method embodiments when the computer program or the instruction is executed by the processor. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0188] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0189] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0190] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0191] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A model training method, characterized in that: include: Based on the structural analysis of the model to be trained, the context is divided into context tasks; Performing dependency analysis on the context tasks to determine a task dependency graph; According to the task dependency graph, the context task is configured with a priority, and the context task with the configured priority is assigned to the corresponding computing node for training in an execution order, and the load information and task execution information of the computing node are obtained; According to the load information and task execution information of the computing nodes, the context tasks are scheduled between the computing nodes, the execution order of the context tasks within the computing nodes is adjusted, and the training results are output.

2. The model training method according to claim 1, characterized in that: Based on the structural analysis of the model to be trained, the context is divided into context tasks, including: Performing structural analysis on the model to be trained to determine a context list; the context list includes multiple contexts; calculating the complexity of the plurality of contexts and dependencies between the plurality of contexts; Each of the multiple contexts is divided into context tasks according to complexity of the multiple contexts and dependencies between the multiple contexts.

3. The model training method according to claim 2, characterized in that: Perform structural analysis on the model to be trained and determine the context list, including: Performing structural analysis on the topological structure of the model to be trained, identifying the components and modules of the model to be trained, and determining the architecture of the model to be trained; According to the architecture of the model to be trained, multiple contexts that are related in sequence are identified to form a context list.

4. The model training method according to claim 1, characterized in that: Performing dependency analysis on the context tasks to determine a task dependency graph includes: Determining the training algorithm properties of the context task; the training algorithm properties of the context task include: forward calculation, back propagation, and gradient update; Classifying the context tasks according to the training algorithm properties of the context tasks to determine task nodes; wherein the task nodes have location information; Analyze the data dependency, calculation order dependency and resource dependency between the task nodes to determine the dependency edge; A directed acyclic graph is drawn according to the dependency edges and the task nodes and their position information to determine a task dependency graph.

5. The model training method according to claim 1, characterized in that: According to the task dependency graph, the context task is configured with a priority, and the context task with the configured priority is assigned to a corresponding computing node for training in an execution order, and load information and task execution information of the computing node are obtained, including: configuring a priority for the context task according to the task dependency graph; Determine the execution order of the context tasks with configured priorities according to different priorities, and organize the context tasks into a task queue according to the execution order; Configuring a load balancing strategy based on the performance information of the computing node; Allocating the task queue to a corresponding computing node according to the load balancing strategy, so that the computing node trains the context tasks in the task queue according to the execution order; Obtain load information and task execution information of the computing node during the training process.

6. The model training method according to claim 5, characterized in that: According to the load information and task execution information of the computing nodes, the context tasks are scheduled between the computing nodes and the execution order of the context tasks in the computing nodes is adjusted, and the training results are output, including: Configure a task scheduling strategy according to the priority of the context task and the execution order of the task queue; According to the load information of the computing node, if it is detected that the load of the computing node exceeds a set threshold, determining the context task to be migrated and the target computing node according to the load balancing strategy; Determining an embedding position of a task queue in the target computing node according to the task scheduling strategy; Migrating the context task to be migrated to the embedded position of the task queue in the target computing node, and determining an updated task queue and an updated execution order; The target computing node continues to perform training on the updated task queue according to the updated execution order; If all computing nodes complete training, the training results are output.

7. The model training method according to claim 6, characterized in that: Also includes: According to the task execution information, if it is detected that there is a context task to be scheduled in the computing node, the priority of the context task to be scheduled is adjusted according to the task scheduling strategy to modify the execution order of the current task queue in which the context task to be scheduled is located; the context task to be scheduled is a context task whose actual execution time exceeds the expected execution time by a set monitoring threshold; Determine, according to the modified execution order of the current task queue and the task dependency graph, an associated task queue having a dependency relationship with the current task queue; The associated task queue and the current task queue are located in the same computing node or in different computing nodes; The execution order of the associated task queues is adjusted according to the task scheduling strategy and the task dependency graph.

8. The model training method according to claim 6, characterized in that: Also includes: According to the task execution information, if it is detected that there is a context task to be scheduled in the computing node, a target scheduling computing node is determined according to a task scheduling strategy and a load balancing strategy; Obtaining the execution order of the task queue in the target scheduling computing node; Determine the embedded position to be scheduled according to the execution order of the task queue in the target scheduling computing node and the task dependency graph; The priority of the target scheduling computing node is modified and scheduled to the embedded position to be scheduled, and the scheduled task queue and the scheduled execution order in the target scheduling computing node are determined.

9. A model training device, characterized in that: include: A context division module, which is used to divide the context into context tasks based on the structural analysis of the model to be trained; A task dependency graph determination module, used to perform dependency analysis on the context tasks and determine a task dependency graph; A context task allocation module is used to configure priorities for the context tasks according to the task dependency graph, and allocate the context tasks with configured priorities to corresponding computing nodes for training in an execution order, and obtain load information and task execution information of the computing nodes; The task scheduling and execution order adjustment module is used to schedule the context tasks between the computing nodes and adjust the execution order of the context tasks within the computing nodes according to the load information and task execution information of the computing nodes, and output the training results.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the model training method described in any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model training method described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the model training method described in any one of claims 1 to 8.

Images