Model training method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a model training method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a model training task and a model structure feature corresponding to the model training task; acquiring a current resource state; dividing the model training task into a plurality of sub-tasks; selecting a target resource scheduling strategy based on the model training task, the model structure feature and the current resource state; and distributing the subtasks to corresponding computing nodes based on the target resource scheduling strategy, wherein the computing nodes are used for executing the corresponding subtasks. According to the embodiment, the target resource scheduling strategy is selected based on the model training task, the model structure feature and the current resource state, the sub-tasks are efficiently allocated to the computing nodes through the target resource scheduling strategy, and the distributed training efficiency and the resource utilization rate are improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence and big data technologies, distributed models are being used more and more widely. In large-scale data centers or cloud computing environments, the training process of distributed models requires efficient resource scheduling and optimization algorithms to enable tasks to be completed quickly and accurately under limited resource conditions.

[0003] Existing distributed model training methods often rely on fixed resource scheduling and load balancing algorithms, resulting in uneven resource utilization, low task execution efficiency, and consequently affecting the overall system performance.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a model training method, apparatus, electronic device, storage medium, and program product, which at least to some extent overcomes the problem of low efficiency in model training tasks due to related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a model training method is provided, comprising: acquiring a model training task and the model structure features corresponding to the model training task; acquiring the current resource state; dividing the model training task into multiple subtasks; selecting a target resource scheduling strategy based on the model training task, the model structure features, and the current resource state; and allocating the subtasks to corresponding computing nodes based on the target resource scheduling strategy, wherein the computing nodes are used to execute the corresponding subtasks.

[0008] In one possible embodiment, the method further includes: during the execution of subtasks by the computing node, obtaining the complexity and execution time of each subtask; calculating the priority of each subtask based on the complexity and execution time of each subtask; and executing the subtasks according to their priorities.

[0009] In one possible implementation, the priority of a subtask is positively correlated with its complexity, and the priority of a subtask is positively correlated with its execution time.

[0010] In one possible embodiment, executing subtasks according to their priority includes: adding each subtask to a task queue in descending order of priority; obtaining the available resources of the computing node; calculating resource suitability based on the available resources of the computing node and the complexity of the subtasks; adjusting the order of the subtasks in the task queue based on the resource suitability; and executing the subtasks in the adjusted order of the subtasks in the task queue.

[0011] In one possible embodiment, the resource suitability of a subtask is positively correlated with the available resources of the computing node, and the resource suitability of a subtask is negatively correlated with the complexity of the subtask.

[0012] In one possible embodiment, the model training task includes input data for the model training task; the model structure features corresponding to the model training task include: the model structure features of the target model; obtaining the model training task and the model structure features corresponding to the model training task includes: obtaining the input data for the model training task; determining the task type of the model training task based on the input data; determining the target model that matches the task type; and obtaining the model structure features of the target model.

[0013] In one possible embodiment, the model structure features include: the distribution of activation functions of each layer in the target model, the number of parameters of each layer in the target model, and the dependencies between modules in the target model; the method further includes: using a target resource scheduling strategy to adjust the computing resources allocated to each layer in the target model based on the distribution of activation functions of each layer in the target model, the number of parameters of each layer in the target model, and the dependencies between modules in the target model.

[0014] In one possible embodiment, the target resource scheduling strategy includes at least one or more of the following: a dynamic scheduling strategy based on reinforcement learning, a scheduling strategy based on quality of service and service level agreement, a scheduling optimization strategy based on container orchestration platform, a scheduling strategy based on edge computing and cloud collaboration, and a scheduling strategy combining heuristic and real-time scheduling.

[0015] In one possible embodiment, the model training task further includes a task requirement for the model training task; the method further includes: determining a learning rate for the model training task based on the task requirement; and calculating the model weights for the next period based on the learning rate of the model training task and the model weights of the target model in the current period.

[0016] In one possible embodiment, the model weights for the next cycle are the sum of the model weights for the current cycle and a first value, wherein the first value is the product of the gradient of the loss function of the target model and the learning rate of the model training task.

[0017] According to another aspect of this disclosure, a model training apparatus is also provided, comprising: a first acquisition module for acquiring a model training task and model structural features corresponding to the model training task; a second acquisition module for acquiring a current resource state; a task decomposition module for dividing the model training task into multiple sub-tasks; a strategy selection module for selecting a target resource scheduling strategy based on the model training task, the model structural features, and the current resource state; and a task allocation module for allocating the sub-tasks to corresponding computing nodes based on the target resource scheduling strategy, wherein the computing nodes are used to execute the corresponding sub-tasks.

[0018] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform a model training method of any of the above via executing the executable instructions.

[0019] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the model training method of any of the above.

[0020] According to another aspect of this disclosure, a computer program product is also provided, comprising: a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement a model training method for any of the above.

[0021] The model training method provided in the embodiments of this disclosure includes: obtaining a model training task and the corresponding model structure features; obtaining the current resource state; dividing the model training task into multiple sub-tasks; selecting a target resource scheduling strategy based on the model training task, model structure features, and current resource state; and allocating the sub-tasks to corresponding computing nodes based on the target resource scheduling strategy, with the computing nodes used to execute the corresponding sub-tasks. This embodiment selects a target resource scheduling strategy based on the model training task, model structure features, and current resource state, and utilizes the target resource scheduling strategy to efficiently allocate sub-tasks to computing nodes, thereby improving distributed training efficiency and resource utilization.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram illustrating an exemplary application system structure of a model training method according to an embodiment of this disclosure is shown.

[0025] Figure 2 This diagram illustrates a flowchart of a model training method according to an embodiment of the present disclosure;

[0026] Figure 3 A flowchart illustrating another model training method in an embodiment of this disclosure is shown;

[0027] Figure 4 A flowchart of yet another model training method in an embodiment of this disclosure is shown;

[0028] Figure 5 This diagram illustrates a flowchart of a subtask execution according to an embodiment of the present disclosure;

[0029] Figure 6 A flowchart of yet another model training method in an embodiment of this disclosure is shown;

[0030] Figure 7 This diagram illustrates a flowchart of the target model determination method in an embodiment of this disclosure;

[0031] Figure 8 A flowchart of yet another model training method in an embodiment of this disclosure is shown;

[0032] Figure 9 This diagram illustrates a model training apparatus according to an embodiment of the present disclosure;

[0033] Figure 10 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] Figure 1 A schematic diagram of an exemplary application system architecture to which the model training methods of the embodiments of this disclosure can be applied is shown. For example... Figure 1 As shown, the system architecture may include terminal device 101, network 102 and server 103.

[0037] Network 102 is a medium used to provide a communication link between terminal device 101 and server 103, and can be a wired network or a wireless network.

[0038] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0039] Terminal device 101 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0040] Optionally, the client of the application installed on different terminal devices 101 may be the same, or the client of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also be different; for example, the application client may be a mobile client, a PC client, etc.

[0041] Server 103 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 101. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.

[0042] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0043] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0044] Under the above system architecture, this disclosure provides a model training method that can be executed by any electronic device with computing power.

[0045] In some embodiments, the model training method provided in this disclosure can be executed by a terminal device of the system architecture described above; in other embodiments, the model training method provided in this disclosure can be executed by a server in the system architecture described above; in still other embodiments, the model training method provided in this disclosure can be implemented by the terminal device and the server in the system architecture described above through interaction.

[0046] Figure 2 This diagram illustrates a model training method flowchart according to an embodiment of the present disclosure, such as... Figure 2 As shown, the model training method provided in this embodiment includes the following steps S202-S210.

[0047] S202. Obtain the model training task and the corresponding model structure features.

[0048] Models can include large distributed models; in other words, a model refers to a large-scale artificial intelligence model that is trained and inferred using a distributed computing architecture. Model training can be understood as a series of operations to train an artificial intelligence model, enabling it to learn patterns in the data and acquire the ability to predict or make decisions about new data.

[0049] Model structural features can be understood as the internal architecture of the model and the properties of its components. For example, model structural features include: the distribution of activation functions in each layer of the target model, the number of parameters in each layer of the target model, and the dependencies between modules in the target model. The target model refers to the model to be trained in the model training task.

[0050] The distribution of activation functions in each layer can be understood as the output statistical characteristics of activation functions in each layer of the neural network, such as mean, variance, peak value, etc. The distribution of activation functions is used to reflect the activity level of the neural network and to determine whether there is a risk of gradient vanishing or exploding in that layer of the neural network.

[0051] The number of parameters in each layer can be understood as the number of parameters in each layer of the neural network. A neural network layer with a large number of parameters has a high computational cost and requires more computing resources, while a neural network layer with a small number of parameters has a low computational cost and requires less computing resources.

[0052] The dependencies between modules can be understood as the execution order of modules in the model. For example, the output of module 1 is the input of module 2, meaning that module 1 must be executed before module 2 can be executed. The dependencies between modules are used to determine the execution order of subtasks and the possibility of parallel processing of subtasks.

[0053] Model structure features also include network architecture types, including but not limited to: Transformer, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), etc. Different architectures have different computation modes, and the resource scheduling strategies used by the computation modules of different architectures are different.

[0054] In one possible implementation, the model training task includes, but is not limited to: the original training dataset, initial model parameters, task configuration parameters, QoS (Quality of Service) and SLA (Service Level Agreement) constraints, task requirements, etc.

[0055] The original training dataset can be understood as the basic data used for model learning. Depending on the application scenario, the form of the training dataset varies, such as text corpora in natural language processing, image data in computer vision, and audio data in audio processing.

[0056] The initial parameters of a model can be understood as the initial weights (W) and biases (b) of each layer of the model, provided by random initialization or pre-training. For example, the initial weight matrix of the attention layer in a Transformer model. These initial parameters serve as the starting point for model training iterations and are gradually optimized through dynamic weight adjustment strategies.

[0057] Task configuration parameters can be understood as metadata describing the attributes of the model training task, including hyperparameters, priority metrics, etc. Hyperparameters include learning rate (η), batch size, training epochs, etc. Priority metrics include task complexity (C), execution time requirement (T), etc.

[0058] QoS / SLA constraints can be understood as the service quality requirements for model training tasks, such as response time thresholds, throughput targets, and error rate caps. For example, an image recognition model requires inference latency to be less than 50ms. When the inference latency is greater than or equal to 50ms during task execution, resource reallocation or priority adjustment is triggered.

[0059] In one possible implementation, obtaining the model training task includes: obtaining the model training task input by the user; determining the target model to be trained based on the model training task; and then obtaining the model structure features of the target model as the model structure features corresponding to the model training task. The model training task carries a target model identifier.

[0060] In one possible implementation, the output data of activation functions at each layer is obtained in real time through hook functions, and statistical measures of the output data are calculated. These statistical measures include, but are not limited to, the mean, variance, and histogram distribution. Alternatively, a non-intrusive monitoring method can be used to obtain the distribution of activation functions at each layer, avoiding significant delays in the model computation process. Furthermore, for large-scale distributed models, the distribution of activation functions across all computing nodes needs to be aggregated through a distributed communication framework.

[0061] In one possible implementation, the number of parameters for each layer is obtained by parsing the model definition file or computation graph. For example, the number of parameters for each layer can be obtained directly using the model.summary() interface, or the total number of parameters can be calculated by traversing the shape of the model weight tensor.

[0062] In one possible implementation, the input-output dependencies between modules are identified by statically analyzing the computational graph of the model. For example, the computational graph can be visualized to indicate the data flow between modules; or a topological sorting algorithm can be used to generate the execution order of modules and determine strong and weak dependencies.

[0063] S204. Get the current resource status.

[0064] The current resource status can be understood as the resource status of a distributed system executing model training tasks. A distributed system refers to a collaborative computing framework composed of multiple computing nodes interconnected through a network. These computing nodes include, but are not limited to, servers, GPU (Graphics Processing Unit) clusters, and edge devices.

[0065] The current resource status can be understood as the resource usage and available resources of each computing node in a distributed system at a given moment. The current resource status includes, but is not limited to, hardware resource status and network resource status. Hardware resource status includes: CPU (Central Processing Unit) utilization, memory usage, remaining storage space, GPU utilization, and I / O (Input / Output) throughput. Network resource status includes: network bandwidth utilization, communication latency, and packet loss rate.

[0066] In one possible implementation, an agent program is deployed on each compute node to collect the hardware resource status of the compute nodes in real time. The agents on each compute node periodically push resource status updates, which are then aggregated to a central node through a distributed communication architecture. Alternatively, network monitoring tools can be used to collect network traffic data between compute nodes, calculating bandwidth utilization, communication latency, and packet loss rate as network resource status.

[0067] S206. Divide the model training task into multiple sub-tasks.

[0068] A subtask is the smallest unit of execution after the model training task has been broken down according to a certain strategy. A subtask can be processed independently by a single computing node or collaboratively with other nodes. Alternatively, a single computing node can execute multiple subtasks. Each subtask undertakes a portion of the computational logic of the model training task. For example, a subtask might be the computational logic of layer 1 in the neural network of the model, or subtask 1 might be the training task for the first batch of data, and subtask 2 might be the training task for the second batch of data.

[0069] In one possible implementation, the model training task is decomposed into multiple subtasks, as represented by Equation (1):

[0070]

[0071] Where T represents the model training task, T i Let represent the i-th subtask. N represents the number of subtasks after decomposition. Executing multiple subtasks in parallel across multiple computing nodes can significantly improve inference speed and reduce system latency.

[0072] In one possible implementation, in a distributed environment, multiple large models with heterogeneous structures can be supported to collaboratively participate in training tasks, automatically matching task types with model strengths and improving overall training accuracy and speed.

[0073] S208. Select a target resource scheduling strategy based on the model training task, model structure features, and current resource status.

[0074] Target resource scheduling strategy refers to the optimal resource allocation and task scheduling strategy dynamically determined based on model training task requirements, model structural characteristics, and current resource status. Target resource scheduling strategies include, but are not limited to: dynamic scheduling strategies based on reinforcement learning, scheduling strategies based on Quality of Service (QoS) and Service Level Agreements (SLAs), scheduling optimization strategies based on container orchestration platforms, scheduling strategies based on edge computing and cloud collaboration, and scheduling strategies combining heuristics and real-time scheduling.

[0075] In one possible implementation, when the computational cost of the model training task changes dynamically, a reinforcement learning-based dynamic scheduling strategy can be selected as the target resource scheduling strategy. For example, dynamic changes in computational cost might include large differences in resource requirements across different training stages, or a model architecture with large parameters such as the Transformer. Reinforcement learning methods can include, but are not limited to, deep Q-networks or policy gradient methods. A reinforcement learning-based dynamic scheduling strategy can be understood as using reinforcement learning to automatically learn and generate the optimal scheduling strategy, taking the current resource state as input.

[0076] In one possible implementation, when the model training task is a high-priority task, a scheduling strategy based on QoS and SLA protocols is used as the target resource scheduling strategy. The scheduling strategy based on QoS and SLA protocols can be understood as setting QoS indicators and SLA requirements for different tasks. By monitoring these QoS and SLA indicators in real time, resource allocation is automatically adjusted or the priority is increased when a task deviates from the preset SLA.

[0077] In one possible implementation, for large-scale distributed training tasks, a scheduling optimization strategy based on a container orchestration platform and a dynamic scheduling strategy based on reinforcement learning are selected as the target resource scheduling strategies. The training tasks are containerized using container orchestration systems such as Kubernetes, enabling dynamic scaling and resource isolation. The scheduling optimization strategy based on a container orchestration platform refers to utilizing the resource scheduling capabilities of the container orchestration system, combined with custom scheduling strategy plugins, to perform containerized deployment and dynamic scheduling of distributed training tasks.

[0078] In one possible implementation, when the distributed system is deployed with heterogeneous nodes, a scheduling strategy based on edge computing and cloud collaboration is selected as the target resource scheduling strategy. Computationally intensive tasks are assigned to cloud GPU nodes, while lightweight tasks are assigned to edge or CPU nodes, achieving collaborative utilization of heterogeneous resources. The scheduling strategy based on edge computing and cloud collaboration refers to dispatching lightweight, latency-sensitive tasks to edge nodes for processing, while concentrating computationally intensive tasks in the cloud or data center for processing, with both working together to optimize overall resource utilization.

[0079] In one possible implementation, when multiple tasks are executed concurrently and the priorities of the tasks differ significantly, a scheduling strategy combining heuristics and real-time scheduling is selected as the target resource scheduling strategy. The target resource scheduling strategy first generates global scheduling candidate schemes using an offline heuristic algorithm, and then dynamically adjusts the task execution order based on real-time monitoring data to ensure timely scheduling of high-priority tasks.

[0080] In this embodiment, diverse optimization logic is used to adapt to the training needs of different scenarios, thereby improving resource utilization and model training efficiency.

[0081] S210. Based on the target resource scheduling strategy, subtasks are allocated to the corresponding computing nodes, and the computing nodes are used to execute the corresponding subtasks.

[0082] A computing node can be understood as a physical or virtual node in a distributed system that has independent computing capabilities, used to execute subtasks and collaborate with other nodes to complete training objectives.

[0083] When the target resource scheduling strategy is a reinforcement learning-based dynamic scheduling strategy, the specific implementation steps of resource scheduling are as follows: define the system state, task allocation actions, and reward functions based on task completion efficiency and resource utilization; repeatedly train the scheduling strategy model using historical data; and select the optimal scheduling scheme in real time based on the current resource state during actual operation. In this embodiment, model structure features are introduced as input to the scheduling strategy model. In addition to the resource dimension, the scheduling strategy model also considers internal dynamic features such as training progress, gradient oscillation values, and parameter change rates, making the scheduling results more globally optimized.

[0084] When the target resource scheduling strategy is a scheduling strategy based on Quality of Service (QoS) and Service Level Agreement (SLA), the specific implementation steps of resource scheduling are as follows: classify subtasks according to response time and accuracy requirements; collect QoS indicator data of subtasks; and adjust the scheduling strategy in a timely manner based on monitoring data so that critical tasks meet SLA requirements.

[0085] When the target resource scheduling strategy is a scheduling optimization strategy based on a container orchestration platform, the specific implementation steps of resource scheduling are as follows: Encapsulate the model training task into a container; develop plugins to schedule tasks based on parameters such as node health status, resource requests, and task priorities; deploy different components of the model on multiple heterogeneous nodes and call them via lightweight RPC or remote tensor references to improve the availability and scalability of large models in resource-constrained edge environments. Implement automatic expansion or contraction of containers to optimize resource allocation.

[0086] When the target resource scheduling strategy is based on edge computing and cloud collaboration, the specific implementation steps of resource scheduling are as follows: Based on the latency requirements and computational intensity of the model training tasks, the model training tasks are divided into edge tasks and cloud tasks; the load and network latency of edge and cloud nodes are monitored in real time; the distribution of sub-tasks is dynamically adjusted based on the monitoring data to achieve optimal collaboration between edge and cloud resources. Based on the computational complexity and bandwidth sensitivity of each model layer, modules with low computational complexity but high interaction frequency are deployed at the edge, while modules with high computational density are deployed in the cloud, optimizing both bandwidth and latency bottlenecks.

[0087] The target resource scheduling strategy is a combination of heuristic and real-time scheduling. The specific implementation steps of resource scheduling are as follows: generate scheduling candidate schemes offline using historical data; fine-tune the candidate schemes based on real-time data during subtask execution; and dynamically adjust the task queue order to achieve optimal allocation of task resources.

[0088] After determining the computing nodes corresponding to each subtask using the target resource scheduling strategy, each subtask is assigned to the corresponding computing node, so that the computing node executes its corresponding subtask.

[0089] In one possible implementation, during the execution of its corresponding subtask by the computing node, the current resource status of the distributed system is periodically obtained, and a target resource scheduling strategy is selected based on the model training task, model structure features and the current resource status of the distributed system. The subtask is then allocated to the corresponding computing node based on the target resource scheduling strategy, so as to realize dynamic adjustment of resources during the execution of the model training task.

[0090] In this embodiment, a target resource scheduling strategy is selected based on the model training task, model structure features, and current resource status. The target resource scheduling strategy is used to efficiently allocate subtasks to computing nodes, thereby improving distributed training efficiency and resource utilization.

[0091] Based on the above embodiments, this embodiment optimizes the model training method, such as... Figure 3 As shown, the optimized model training method also includes steps S302-S306.

[0092] S302. During the execution of multiple subtasks on the computing node, obtain the complexity and execution time of each subtask.

[0093] The complexity of a subtask can be understood as a comprehensive indicator of its difficulty in terms of computation and resource consumption. It is used to assess the resource requirements of a subtask on computing nodes. Higher complexity requires more computing power, memory, or network resources. The execution time of a subtask can be understood as the cumulative time elapsed from the start of execution to the current moment, reflecting the subtask's progress and urgency.

[0094] In one possible implementation, the complexity of a subtask is determined by the number of parameters in the model layer corresponding to that subtask. The larger the number of parameters in the model layer corresponding to a subtask, the higher the complexity of that subtask. For example, the attention layer of a Transformer typically accounts for 30% to 50% of the total number of parameters in the model, and the subtasks corresponding to it have higher complexity than other model layers.

[0095] In one possible implementation, different activation functions have different computational complexities, and the complexity of the subtask is evaluated by monitoring the activation function type and output distribution. For example, the Sigmoid function requires exponential operations, making it more complex than the ReLU function.

[0096] In one possible implementation, the complexity of a subtask is dynamically assessed by monitoring metrics such as CPU utilization, memory usage, and GPU computing power consumption on the computing nodes. For example, if subtask 1 consistently consumes more than 80% of the GPU computing power, it indicates high computational complexity; if subtask 1 consistently consumes only 10% of the GPU computing power, it indicates low computational complexity.

[0097] In one possible implementation, the longer the single iteration time of a subtask, the higher the complexity. For example, if the iteration time of the backpropagation subtask is 50% longer than that of the forward propagation subtask, the complexity may be higher due to gradient calculation.

[0098] In one possible implementation, a start timestamp is recorded when the subtask starts, and the current timestamp is obtained after execution or periodically monitored. The difference between the current timestamp and the start timestamp is calculated as the execution time of the subtask.

[0099] S304. Calculate the priority of each subtask based on its complexity and execution time. The priority of a subtask is positively correlated with its complexity, and the priority of a subtask is positively correlated with its execution time.

[0100] The priority of a subtask can be understood as a scheduling priority calculated based on its complexity and execution time, used to determine the order in which tasks are allocated resources. Subtasks with higher priority receive computing resources from computing nodes first.

[0101] The positive correlation between subtask priority and subtask complexity can be understood as follows: the priority of a subtask increases with its complexity. In other words, the more complex the subtask, the higher its priority in resource scheduling. High-complexity subtasks require more computing resources during execution. The positive correlation between priority and complexity allows high-complexity subtasks to preferentially obtain GPU nodes or high-bandwidth networks, preventing a decrease in training efficiency due to insufficient resources.

[0102] The positive correlation between subtask priority and subtask execution can be understood as follows: the priority of a subtask increases as its execution time increases; that is, the longer the execution time of a subtask, the higher its scheduling priority. Tasks with long execution times often face the risk of timeout. The positive correlation between priority and execution time allows the subtask to be completed first, preventing it from being indefinitely postponed. For example, if the execution time of the inference subtask in a risk control model approaches the 50ms threshold, its priority automatically increases to trigger resource preemption and prevent it from timeout.

[0103] In this embodiment, the priority calculation formula is shown in formula (2).

[0104] P t =α*C+β*T (2)

[0105] Among them, P t C represents the priority of the subtask, T represents the execution time of the subtask, and α and β are adjustment coefficients that can be set according to the actual situation. They are not limited in this embodiment.

[0106] S306. Execute subtasks in order of priority.

[0107] Subtasks are sorted from highest to lowest priority, and higher-priority subtasks are executed first, ensuring that more complex and time-consuming subtasks receive computing resources first. For example, highly complex and time-consuming subtasks (such as training the core layers of a Transformer model) have the highest priority and are given priority in using GPU computing resources; low-complexity and time-consuming subtasks (such as data preprocessing tasks) have lower priority and are executed when resources are available.

[0108] By calculating and sorting the priorities of subtasks, the problem of uneven resource allocation in traditional static scheduling is solved, thereby improving model training efficiency and resource utilization.

[0109] In one possible implementation, this embodiment optimizes the model optimization method, such as... Figure 4 As shown, the optimized model optimization method mainly includes steps S402-S414.

[0110] S402. During the execution of multiple subtasks on the computing node, obtain the complexity and execution time of each subtask.

[0111] S404. Calculate the priority of each subtask based on its complexity and execution time.

[0112] In this embodiment, the execution flow of S402-S404 is the same as that of S302-S304 in the above embodiment. For details, please refer to the description in the above embodiment. In this embodiment, it will not be repeated.

[0113] S406. Add each subtask to the task queue in descending order of priority.

[0114] A task queue can be understood as a data structure used to store subtasks to be executed. Subtasks are sorted from highest to lowest priority, ensuring that higher-priority tasks are executed first. The queue supports dynamic updates; the queue order is automatically adjusted when new subtasks are added or the priority of existing subtasks changes.

[0115] When using a max-heap, compare their priorities P. t Value and priority P of subtasks in the queue t The value is automatically inserted into the corresponding position so that the highest priority subtask is always at the head of the queue.

[0116] S408. Obtain the available resources of the compute node.

[0117] Available resources can be understood as the total amount of unused resources on a computing node at a given moment. Available resources include available hardware resources and available network resources. Available resources are used to evaluate the computing node's capacity to handle subtasks. Available hardware resources include, but are not limited to: the number of idle CPU cores, remaining memory capacity, and unused GPU computing power. Available network resources include, but are not limited to: available bandwidth and communication latency.

[0118] The method for obtaining available resources is the same as the method for obtaining the current resource status in the above embodiments, and can be referred to the description in the above embodiments for details.

[0119] S410. Calculate resource fit based on the available resources of the computing node and the complexity of the subtask. The resource fit of a subtask is positively correlated with the available resources of the computing node, and negatively correlated with the complexity of the subtask.

[0120] Resource fit can be understood as a quantitative indicator used to evaluate the degree to which the complexity of a subtask matches the available resources of a computing node. The higher the resource fit, the better the available resources of the computing node can meet the resource requirements of the subtask, and the higher the expected task execution efficiency.

[0121] In one possible implementation, resource adaptability is positively correlated with available resources. This can be understood as the more available resources a computing node has, the less likely a subtask will encounter resource bottlenecks when executing on that computing node, and the higher the resource adaptability.

[0122] The negative correlation between resource adaptability and complexity can be understood as follows: the higher the complexity of a subtask, the more stringent its resource requirements. If the available resources remain unchanged, an increase in complexity will lead to a decrease in resource adaptability.

[0123] By quantifying the matching relationship between resources and tasks, the problem of inefficient resource allocation in traditional scheduling is solved, thereby improving model training efficiency and resource utilization.

[0124] In this embodiment, the formula for calculating resource adaptability is shown in formula (3).

[0125]

[0126] Among them, S i For the resource fit of the i-th subtask, R available C represents the available resources for computing nodes. i Let S be the complexity of the subtask. Based on resource suitability S... i The value is used to dynamically adjust the execution order of subtasks in the task queue.

[0127] Tasks are divided into multiple levels based on their complexity. Low-complexity tasks are executed first during low-load periods, while high-complexity tasks are executed when resources are sufficient. This ensures a balanced execution of tasks of different complexities and improves overall throughput.

[0128] Resource fit is introduced as a scheduling reference. The fit is dynamically calculated based on the coupling degree between the available resources of the computing node and the current structural complexity of the model (the complexity of the subtask). Model resources with high structural matching degree are allocated first to reduce overfitting or underfitting.

[0129] S412. Adjust the order of subtasks in the task queue based on resource adaptability.

[0130] The priority of subtasks in the task queue is dynamically adjusted based on their resource adaptability on each computing node. For example, tasks with high adaptability are prioritized at the front of the task queue to ensure rapid resource acquisition; tasks with low adaptability are prioritized at the back of the task queue to delay execution or wait for resource release.

[0131] In one possible implementation, the suitability of all subtasks on each compute node is periodically recalculated, and the queue order is updated. For example, if a node adds GPU resources, the suitability of related subtasks improves, and their queue positions move forward. Alternatively, the queue can be immediately reordered when the following events occur: a change in subtask complexity; a sudden change in the availability of compute node resources; a new subtask being added to the queue; or an existing subtask completing its execution.

[0132] S414. Execute subtasks in the adjusted order of the task queue.

[0133] The adjusted subtask order can be understood as the order of subtasks in the task queue not being fixed, but dynamically adjusted based on resource adaptability.

[0134] like Figure 5 As shown, this embodiment provides a schematic diagram of a task execution process, which, combined with a distributed large model training scenario, mainly includes:

[0135] S502, Enter the task queue

[0136] The newly generated subtasks are added to a task queue to await scheduling. For example, during the training of a large model, subtasks such as calculating the gradient of the attention layer and updating the weight parameters are split and then placed into a unified task queue to await resource allocation by the system.

[0137] S504, Sort by priority

[0138] Tasks in the task queue are arranged in order of priority. The priority can be calculated using formula (2).

[0139] S506, Resource Allocation and Execution

[0140] The sorted subtasks are allocated computing resources in sequence, and the subtask logic is executed after the computing resources are obtained.

[0141] In this embodiment, adjusting the order of subtasks in the task queue according to resource suitability can reduce the waiting time of high-suitability tasks, prevent low-suitability tasks from occupying high-quality resources, and improve resource utilization and model training efficiency.

[0142] In one possible implementation, this embodiment optimizes the model training method, such as... Figure 6 As shown, the optimized model training method includes steps S602-S616.

[0143] S602. Obtain the input data for the model training task.

[0144] The model training task includes the input data for the model training task.

[0145] Input data can be understood as raw or preprocessed data used to train large models, and its type and scale depend on the specific application scenario. For example, in natural language processing tasks, input data may include text corpora; in computer vision tasks, input data may include image or video datasets.

[0146] In one possible implementation, obtaining input data for model training tasks can include: acquiring input data from an internal database in real-time or in batches using ETL (Extract, Transform, Load) tools, or acquiring input data from public datasets or third-party data services via API interfaces or data downloads. For example, autonomous driving models acquire image data from sensors in real-time.

[0147] S604. Determine the task type of the model training task based on the input data.

[0148] Task types can be understood as categories of tasks based on training objectives, input data features, etc. Task types include, but are not limited to, image classification, text sentiment analysis, and time series prediction.

[0149] In one possible implementation, the data characteristics of the input data are analyzed. If the input data consists of multiple images labeled with categories, the task type is determined to be image classification based on common task knowledge. If the input data is text with sentiment bias, it is determined to be text sentiment analysis. Alternatively, data metadata can be used to assist in the determination; for example, if the data is labeled "for temperature prediction," the task type may be time series prediction. Furthermore, user configuration can be incorporated; if the user specifies "perform image recognition" when submitting the task, the task type is directly determined to be image recognition.

[0150] S606. Determine the target model that matches the task type.

[0151] The target model can be understood as a model architecture that is adapted to the task type and effectively completes the training task of that type. For example, convolutional neural networks (CNNs) are commonly used for image classification, and Transformers are commonly used for natural language processing.

[0152] In one possible implementation, a mapping rule base between task types and models is established. Once the task type is determined, the rule base is queried to select a matching model as the target model. If the image classification task has high real-time requirements, a lightweight and fast inference model is selected from CNN models based on the mapping rules as the target model; if high accuracy is desired, a high-performance ResNet-50 model can be selected as the target model.

[0153] In one possible implementation, to improve the adaptability of the large model to different task scenarios, this embodiment introduces a multi-layered model layering mechanism. By decomposing the large model into multiple sub-model layers with different functions, an appropriate model layer is selected for inference based on the type of different tasks.

[0154] like Figure 7 As shown, the main steps include the following:

[0155] S702, Task Type Recognition

[0156] Based on the characteristics of the input data, the task type is identified using a pre-trained classifier.

[0157] S704, Model Level Selection

[0158] The optimal model layer is matched as the target model based on the task type. For simple tasks, a shallow model is selected as the target model; for complex tasks, a deep model is selected as the target model to ensure accuracy.

[0159] S706, Execution Reasoning

[0160] After selecting a suitable target model, the system enters the inference phase and returns the results.

[0161] In one possible implementation, under conditions of resource scarcity or high real-time requirements, a lightweight version of the model structure can be automatically activated based on the pruning graph generated during the training phase. For example, knowledge distillation, low-rank decomposition, or dynamic neural network pruning strategies can be used to reduce the overhead of training or inference tasks while keeping performance loss under control.

[0162] S608. Obtain the model structure features of the target model.

[0163] The model structure features corresponding to the model training task include: the model structure features of the target model; the model structure features include: the distribution of activation functions in each layer of the target model, the number of parameters in each layer of the target model, and the dependencies between modules in the target model.

[0164] The distribution of activation functions in each layer can be understood as the statistical characteristics of the output of activation functions in each layer of the neural network, such as mean, variance, and peak value. The distribution of activation functions reflects the activity level of neurons and can be used to determine whether there is a risk of vanishing or exploding gradients. The number of parameters in each layer can be understood as the number of parameters in each layer of the neural network. A neural network layer with a large number of parameters has a high computational cost and requires more computing resources. The dependencies between modules can be understood as the execution order of modules in the model. For example, the output of module 1 is the input of module 2, meaning that module 2 can only be executed after module 1 has completed. The dependencies between modules are used to determine the execution order of subtasks and the possibility of parallel processing.

[0165] In this embodiment, the task type is first identified by input data, then the corresponding target model is matched, and finally the structural features corresponding to the target model are obtained, so as to realize targeted training strategy design, optimize training resource allocation, and improve training efficiency.

[0166] S610: Obtain the current resource status of the distributed system.

[0167] S612. Divide the model training task into multiple sub-tasks.

[0168] S614. Select a target resource scheduling strategy based on the model training task, model structure features, and the current resource status of the distributed system.

[0169] The execution flow of S610-S614 in this embodiment is the same as that of S204-S208 in the above embodiment. For details, please refer to the description in the above embodiment.

[0170] S616. Utilize the target resource scheduling strategy to adjust the computing resources allocated to each layer in the target model based on the distribution of activation functions in each layer of the target model, the number of parameters in each layer of the target model, and the dependencies between modules in the target model.

[0171] The statistical distribution of activation functions in each layer of the neural network is monitored in real time, and resource allocation is adjusted according to the distribution characteristics of activation functions to avoid training instability caused by activation function saturation or gradient vanishing.

[0172] During training, the number of parameters for each layer is recorded in real time. For layers with a large number of parameters, more computing resources are allocated in advance to ensure efficient execution of the training process.

[0173] By analyzing the dependencies between different modules, we can predict computational bottlenecks and optimize resource scheduling and task allocation, thereby ensuring that critical modules are prioritized for scheduling.

[0174] In this embodiment, the ability to perceive the internal structural features of a large model is introduced to optimize resource allocation and model performance during the training process, thereby improving model training efficiency and resource utilization.

[0175] Based on the above embodiments, this embodiment optimizes the model training method, such as... Figure 8 As shown, the optimization steps include S802-S806.

[0176] S802, Obtain the task requirements for model training.

[0177] Task requirements refer to the specific requirements for task objectives, performance indicators, resource constraints, etc. during model training. They typically include task type, accuracy target, real-time requirements, and resource constraints.

[0178] Obtain task requirements for model training tasks, including extracting task requirements directly from user-submitted training configurations, or automatically populating default task requirements based on requirement templates for similar historical tasks.

[0179] S804. Learning rate determined based on task requirements and model training tasks.

[0180] The learning rate is a hyperparameter in deep learning optimization algorithms that controls the magnitude of model weight updates. It determines the step size for parameter updates in each iteration. An excessively large learning rate may cause the model to fail to converge, while an excessively small learning rate will prolong training time.

[0181] In one possible implementation, a mapping relationship is established between task requirements and learning rates. For example, for real-time priority tasks, the learning rate is automatically increased by 20% to 50% to accelerate weight iteration; for tasks requiring high precision, the learning rate is reduced to 50% to 80% of the baseline value to reduce fluctuations during convergence. After determining the task requirements for model training, the learning rate for the model training task is found in the above mapping relationship.

[0182] S806. Based on the learning rate of the model training task and the model weights of the target model in the current period, calculate the model weights for the next period. The model weights for the next period are the sum of the model weights in the current period and a first value, where the first value is the product of the gradient of the loss function of the target model and the learning rate of the model training task.

[0183] Model weights are the strength values ​​of neuron connections in a neural network, and they are a concrete manifestation of the model's learning ability. For example, the weights of a Transformer model include the weights of the attention layer and the weights of the feedforward neural network. The model weights for the current epoch can be understood as the weight parameters of the target model at the end of the current training epoch.

[0184] The model weights for the next cycle can be understood as the weight parameters for the next training cycle calculated using the model weights, learning rate, and loss function gradient of the current cycle. The model weights for the next cycle are the result of model parameter updates, reflecting the optimization direction of the model during training.

[0185] The loss function of the target model is a function used to measure the difference between the target model's prediction and the true label. The smaller the loss function value, the higher the model's prediction accuracy, and its gradient indicates the optimal direction for weight optimization.

[0186] The loss function L is calculated using the backpropagation algorithm, along with the current cycle model weights W. T gradient This gradient represents the rate at which the loss function changes with the weights, pointing in the direction of the fastest decrease in loss.

[0187] By identifying key parameters through loss sensitivity analysis, and dynamically adjusting the learning rate, higher update priority is given to key weight regions, thereby improving convergence efficiency and reducing unnecessary computational redundancy.

[0188] By calculating the gradient of the loss function with respect to each model weight, we can determine which model weights are sensitive to changes in the loss function, thus identifying the parameters most important for model training. For the model weights identified as critical, their learning rates are increased to enable faster updates during training; while for less important model weights, their update frequency is reduced to conserve computational resources.

[0189] By assigning higher update priority to key parameters through loss sensitivity analysis, the overall training process can be optimized and redundant computations can be reduced.

[0190] The formula for calculating the model weights is shown in formula (4).

[0191]

[0192] Among them, W t+1 W represents the model weights for the next period. T This represents the model weights for the current period, and η represents the learning rate. This represents the gradient of the loss function.

[0193] Higher priority tasks are assigned higher learning rates, enabling their weights to respond quickly to new data features and improving model training efficiency.

[0194] In this embodiment, the model weights are dynamically adjusted according to task requirements to improve the accuracy and efficiency of task processing.

[0195] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution comply with the relevant provisions of laws and regulations. All types of data, such as personal identity data, operational data, and behavioral data related to individuals, customers, and groups, obtained in this disclosed embodiment have been agreed upon by the users.

[0196] Based on the same inventive concept, this disclosure also provides a model training device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0197] Figure 9 This diagram illustrates a model training apparatus according to an embodiment of the present disclosure, such as... Figure 9 As shown, the device includes: a first acquisition module 910, a second acquisition module 920, a task decomposition module 930, a strategy selection module 940, and a task allocation module 950.

[0198] The system includes a first acquisition module 910 for acquiring a model training task and the corresponding model structure features; a second acquisition module 920 for acquiring the current resource status; a task decomposition module 930 for dividing the model training task into multiple subtasks; a strategy selection module 940 for selecting a target resource scheduling strategy based on the model training task, the model structure features, and the current resource status; and a task allocation module 950 for allocating the subtasks to corresponding computing nodes based on the target resource scheduling strategy, wherein the computing nodes are used to execute the corresponding subtasks.

[0199] In one possible embodiment, the system further includes: a priority processing module, configured to obtain the complexity and execution time of each subtask during the execution of subtasks on the computing node; calculate the priority of each subtask based on the complexity and execution time of each subtask; and execute the subtasks according to their priorities.

[0200] In one possible implementation, the priority of a subtask is positively correlated with its complexity, and the priority of a subtask is positively correlated with its execution time.

[0201] In one possible embodiment, the priority processing module is further configured to add each subtask to the task queue in descending order of priority; obtain the available resources of the computing node; calculate the resource suitability based on the available resources of the computing node and the complexity of the subtask; adjust the order of the subtasks in the task queue based on the resource suitability; and execute the subtasks in the adjusted order of the subtasks in the task queue.

[0202] In one possible embodiment, the resource suitability of a subtask is positively correlated with the available resources of the computing node, and the resource suitability of a subtask is negatively correlated with the complexity of the subtask.

[0203] In one possible embodiment, the model training task includes input data for the model training task; the model structure features corresponding to the model training task include: the model structure features of the target model; the first acquisition module 910 is specifically used to acquire the input data for the model training task; determine the task type of the model training task based on the input data; determine the target model that matches the task type; and acquire the model structure features of the target model.

[0204] In one possible embodiment, the model structure features include: the distribution of activation functions of each layer in the target model, the number of parameters of each layer in the target model, and the dependencies between modules in the target model; the device further includes: a resource scheduling module, used to adjust the computing resources allocated to each layer in the target model based on the distribution of activation functions of each layer in the target model, the number of parameters of each layer in the target model, and the dependencies between modules in the target model using a target resource scheduling strategy.

[0205] In one possible embodiment, the target resource scheduling strategy includes at least one or more of the following: a dynamic scheduling strategy based on reinforcement learning, a scheduling strategy based on quality of service and service level agreement, a scheduling optimization strategy based on container orchestration platform, a scheduling strategy based on edge computing and cloud collaboration, and a scheduling strategy combining heuristic and real-time scheduling.

[0206] In one possible embodiment, the model training task further includes task requirements for the model training task; the device further includes: a weight dynamic adjustment module, used to determine the learning rate of the model training task based on the task requirements; and to calculate the model weights for the next period based on the learning rate of the model training task and the model weights of the target model in the current period.

[0207] In one possible embodiment, the model weights for the next cycle are the sum of the model weights for the current cycle and a first value, wherein the first value is the product of the gradient of the loss function of the target model and the learning rate of the model training task.

[0208] It should be noted that the examples and application scenarios implemented by the modules in the above device embodiments and the corresponding steps in the method embodiments are the same, but are not limited to the content disclosed in the above method embodiments. It should also be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0209] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which can be collectively referred to herein as a "circuit", "module" or "system".

[0210] Based on the same inventive concept, this disclosure also provides an electronic device, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the model training method of any of the above-described methods by executing the executable instructions. Since the principle by which this electronic device solves the problem is similar to that of the above-described method embodiments, the implementation of this electronic device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0211] The following reference Figure 10 To describe an electronic device 1000 according to such an embodiment of the present disclosure. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0212] like Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, and a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010).

[0213] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1010 can perform the following steps of the above method embodiments: obtaining a model training task and the model structure features corresponding to the model training task; obtaining the current resource status; dividing the model training task into multiple sub-tasks; selecting a target resource scheduling strategy based on the model training task, the model structure features, and the current resource status; and allocating the sub-tasks to corresponding computing nodes based on the target resource scheduling strategy, wherein the computing nodes are used to execute the corresponding sub-tasks.

[0214] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache 10202, and may further include read-only memory (ROM) 10203.

[0215] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0216] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0217] Electronic device 1000 can also communicate with one or more external devices 1040 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0218] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network-side device, etc.) to execute the method according to the embodiments of this disclosure.

[0219] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method described above. Since the principle by which this computer-readable storage medium embodiment solves the problem is similar to that of the above method embodiments, the implementation of this computer-readable storage medium embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0220] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0221] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0222] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0223] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0224] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the model training method of any one of the above method embodiments. Since the principle by which this computer program product embodiment solves the problem is similar to that of the above method embodiments, the implementation of this computer program product embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0225] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0226] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0227] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network-side device, etc.) to execute the methods according to the embodiments of this disclosure.

[0228] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A model training method, characterized in that, include: Obtain the model training task and the corresponding model structure features; Get the current resource status; The model training task is divided into multiple sub-tasks; A target resource scheduling strategy is selected based on the model training task, the model structural features, and the current resource status. Based on the target resource scheduling strategy, the subtasks are allocated to the corresponding computing nodes, and the computing nodes are used to execute the corresponding subtasks.

2. The model training method according to claim 1, characterized in that, The method further includes: During the execution of multiple subtasks on the computing node, the complexity and execution time of each subtask are obtained. The priority of each subtask is calculated based on its complexity and execution time. The subtasks are executed according to their respective priorities.

3. The model training method according to claim 2, characterized in that, The priority of a subtask is positively correlated with its complexity, and the priority of a subtask is positively correlated with its execution time.

4. The model training method according to claim 3, characterized in that, The step of executing the subtasks according to their respective priorities includes: Each of the subtasks is added to the task queue in descending order of priority. Obtain the available resources of the computing node; The resource suitability is calculated based on the available resources of the computing node and the complexity of the subtask. The order of the subtasks in the task queue is adjusted based on the resource suitability. The subtasks are executed in the adjusted order of the subtasks in the task queue.

5. The model training method according to claim 4, characterized in that, The resource adaptability of the subtask is positively correlated with the available resources of the computing node, and the resource adaptability of the subtask is negatively correlated with the complexity of the subtask.

6. The model training method according to claim 1, characterized in that, The model training task includes the input data for the model training task; The model structure features corresponding to the model training task include: the model structure features of the target model; The acquisition of the model training task and the corresponding model structure features include: Obtain the input data for the model training task; The task type of the model training task is determined based on the input data; Determine the target model that matches the task type; Obtain the model structure features of the target model.

7. The model training method according to claim 6, characterized in that, The model structure features include: the distribution of activation functions in each layer of the target model, the number of parameters in each layer of the target model, and the dependencies between modules in the target model; The method further includes: The target resource scheduling strategy adjusts the computing resources allocated to each layer of the target model based on the distribution of activation functions of each layer, the number of parameters of each layer of the target model, and the dependencies between modules of the target model.

8. The model training method according to claim 6, characterized in that, The target resource scheduling strategy includes at least one or more of the following: dynamic scheduling strategy based on reinforcement learning, scheduling strategy based on quality of service and service level agreement, scheduling optimization strategy based on container orchestration platform, scheduling strategy based on edge computing and cloud collaboration, and scheduling strategy combining heuristic and real-time scheduling.

9. The model training method according to claim 6, characterized in that, The model training task also includes the task requirements for the model training task; the method also includes: The learning rate for the model training task is determined based on the task requirements. The model weights for the next period are calculated based on the learning rate of the model training task and the model weights of the target model in the current period.

10. The model training method according to claim 9, characterized in that, The model weight for the next cycle is the sum of the model weight for the current cycle and a first value, wherein the first value is the product of the gradient value of the loss function of the target model and the learning rate of the model training task.

11. A model training device, characterized in that, include: The first acquisition module is used to acquire the model training task and the model structure features corresponding to the model training task; The second acquisition module is used to acquire the current resource status; The task decomposition module is used to divide the model training task into multiple sub-tasks. The strategy selection module is used to select a target resource scheduling strategy based on the model training task, the model structural features, and the current resource status. The task allocation module is used to allocate the subtasks to the corresponding computing nodes based on the target resource scheduling strategy, and the computing nodes are used to execute the corresponding subtasks.

12. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the model training method of any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1 to 10.

14. A computer program product comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the model training method according to any one of claims 1 to 10.

Images