Model training methods, devices, electronic equipment, servers, and storage media

Abstract

This disclosure proposes a model training method, apparatus, electronic device, server, and storage medium, applied to a resource manager that manages the resources of a server cluster. The method includes: acquiring configuration information, wherein the configuration information is determined by information about the model to be trained; determining available resource information of the server cluster; determining a first computing server from multiple computing servers based on the configuration information and available resource information, wherein the multiple computing servers belong to the server cluster; and generating a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained. During model training, this effectively supports the elastic scaling of nodes in the server cluster, enabling flexible allocation of server cluster resources, effectively avoiding resource waste, and improving model training efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, apparatus, electronic device, server, and storage medium. Background Technology

[0002] As the complexity of models performing artificial intelligence tasks increases and the scale of data grows, distributed model training based on server clusters has become the mainstream training method. In distributed training, by integrating the resources of multiple server nodes, parallel execution of model training tasks can be achieved, thereby improving model training efficiency. In distributed model training scenarios, the allocation of model training tasks among the server nodes in the server cluster, as well as the interaction between server nodes, exhibit corresponding architectural patterns, such as the parameter server architecture.

[0003] In related technologies, in a parameter server architecture, some nodes are assigned as parameter servers, which are used to split and store model parameters and are responsible for updating the parameters; other nodes are designated as computing servers, which are used to read training data in parallel and train the model, periodically push gradients to the parameter servers and pull the latest model parameters for model training.

[0004] In this approach, elastic scaling of nodes in the server cluster is not supported during model training, making it impossible to flexibly allocate server cluster resources, which can easily lead to resource waste and affect model training efficiency. Summary of the Invention

[0005] This disclosure aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the purpose of this disclosure is to propose a model training method, apparatus, electronic device, server, storage medium, and computer program product that effectively supports the elastic scaling of nodes in a server cluster during model training, enables flexible allocation of server cluster resources, effectively avoids resource waste, and improves model training efficiency.

[0007] The model training method proposed in the first aspect of this disclosure is applied to a resource manager that manages the resources of a server cluster. The method includes: obtaining configuration information, wherein the configuration information is determined by information about the model to be trained; determining available resource information of the server cluster; determining a first computing server from multiple computing servers based on the configuration information and available resource information, wherein the multiple computing servers belong to the server cluster; and generating a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained.

[0008] The model training method proposed in the first aspect of this disclosure obtains configuration information, determines the available resource information of the server cluster, and determines a first computing server from multiple computing servers based on the configuration information and available resource information, and generates a model training task corresponding to the first computing server. Since the configuration information is flexibly determined based on the information of the model to be trained, and the first computing server in the server cluster is flexibly selected and allocated based on the configuration information and the available resource information of the server cluster to execute the model training task based on the first computing server, it can effectively support the elastic scaling of nodes in the server cluster during the model training process, realize the flexible allocation of server cluster resources, effectively avoid resource waste, and improve model training efficiency.

[0009] The model training method proposed in the second aspect of this disclosure is applied to a server cluster, the server cluster including: multiple computing servers, comprising: providing a model to be trained to a resource manager, wherein the model to be trained information is used to determine configuration information; providing the resource manager with available resource information of the server cluster; determining a first computing server based on a model training task generated by the resource manager, wherein the first computing server is determined by the resource manager from multiple computing servers based on the configuration information and available resource information; and controlling the first computing server to train the model to be trained based on the model training task.

[0010] The model training method proposed in the second aspect of this disclosure provides the resource manager with information about the model to be trained and the available resources of the server cluster. Based on the model training task generated by the resource manager, a first computing server is determined and controlled. The model to be trained is then trained based on the model training task. Since the resource manager is provided with information about the model to be trained and the available resources of the server cluster, the resource manager can flexibly select and allocate the first computing server in the server cluster based on this configuration information and the available resources of the server cluster. When the model training task is executed based on the first computing server, the elastic scaling of nodes in the server cluster can be effectively supported during the model training process. This enables flexible allocation of server cluster resources, effectively avoids resource waste, and improves model training efficiency.

[0011] The model training apparatus proposed in the third aspect of this disclosure is applied to a resource manager that manages the resources of a server cluster. The apparatus includes: an acquisition module for acquiring configuration information, wherein the configuration information is determined by information about the model to be trained; a first determination module for determining available resource information of the server cluster; a second determination module for determining a first computing server from multiple computing servers based on the configuration information and available resource information, wherein the multiple computing servers belong to the server cluster; and a generation module for generating a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained.

[0012] The model training apparatus proposed in the third aspect of this disclosure obtains configuration information, determines the available resource information of the server cluster, determines a first computing server from multiple computing servers based on the configuration information and available resource information, and generates a model training task corresponding to the first computing server. Since the configuration information is flexibly determined based on the information of the model to be trained, and the first computing server in the server cluster is flexibly selected and allocated based on the configuration information and the available resource information of the server cluster to execute the model training task based on the first computing server, it can effectively support the elastic scaling of nodes in the server cluster during the model training process, realize the flexible allocation of server cluster resources, effectively avoid resource waste, and improve model training efficiency.

[0013] The model training apparatus proposed in the fourth aspect of this disclosure is applied to a server cluster, the server cluster including: multiple computing servers, including: a first providing module, used to provide a resource manager with information about a model to be trained, wherein the information about the model to be trained is used to determine configuration information; a second providing module, used to provide the resource manager with information about available resources of the server cluster; a seventh determining module, used to determine a first computing server based on a model training task generated by the resource manager, wherein the first computing server is determined by the resource manager from multiple computing servers based on configuration information and available resource information; and a second control module, used to control the first computing server to train the model to be trained based on the model training task.

[0014] The model training apparatus proposed in the fourth aspect of this disclosure provides the resource manager with information about the model to be trained and information about the available resources of the server cluster. Based on the model training task generated by the resource manager, a first computing server is determined and controlled. The model to be trained is trained based on the model training task. Since the resource manager is provided with information about the model to be trained and information about the available resources of the server cluster, the resource manager can flexibly select and allocate the first computing server in the server cluster based on the configuration information and the available resources of the server cluster. When the model training task is executed based on the first computing server, the elastic scaling of nodes in the server cluster can be effectively supported during the model training process, and the flexible allocation of server cluster resources can be realized, effectively avoiding resource waste and improving model training efficiency.

[0015] In a fifth aspect, this disclosure provides an electronic device, including: a model training apparatus as proposed in the embodiments of the third aspect of this disclosure.

[0016] In a sixth aspect, this disclosure provides a server, including: the model training apparatus proposed in the embodiments of the fourth aspect of this disclosure.

[0017] A seventh aspect of this disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a model training method according to a first aspect of this disclosure, or to perform a model training method according to a second aspect of this disclosure.

[0018] In an eighth aspect, this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute a model training method according to an embodiment of a first aspect of this disclosure, or to execute a model training method according to an embodiment of a second aspect of this disclosure.

[0019] In a ninth aspect, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the model training method of the first aspect of this disclosure, or implements the model training method of the second aspect of this disclosure.

[0020] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 This is a schematic flowchart of a model training method proposed in an embodiment of this disclosure;

[0023] Figure 2 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure;

[0024] Figure 3 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure;

[0025] Figure 4 This is a schematic diagram of model training control proposed in another embodiment of this disclosure;

[0026] Figure 5 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure;

[0027] Figure 6 This is a schematic diagram of a model training architecture proposed in another embodiment of this disclosure;

[0028] Figure 7 This is a schematic flowchart of a model training method proposed in an embodiment of this disclosure;

[0029] Figure 8 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure;

[0030] Figure 9 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure;

[0031] Figure 10 This is a schematic diagram of a data distribution server according to another embodiment of this disclosure;

[0032] Figure 11 This is a schematic diagram of a model training scheme proposed in another embodiment of this disclosure;

[0033] Figure 12 This is a schematic diagram of the structure of a model training device proposed in an embodiment of this disclosure;

[0034] Figure 13 This is a schematic diagram of the structure of a model training device proposed in another embodiment of this disclosure;

[0035] Figure 14 This is a schematic diagram of the structure of a model training device proposed in an embodiment of this disclosure;

[0036] Figure 15 This is a schematic diagram of the structure of a model training device proposed in another embodiment of this disclosure;

[0037] Figure 16 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure;

[0038] Figure 17 This is a schematic diagram of the structure of a server according to an embodiment of this disclosure;

[0039] Figure 18 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0040] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0041] Figure 1 This is a schematic flowchart of a model training method proposed in one embodiment of this disclosure.

[0042] It should be noted that the execution subject of the model training method in this embodiment is a model training device, which can be implemented by software and / or hardware. The model training device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0043] The model training method in this embodiment can be applied to a resource manager, which manages the resources of the server cluster. The resource manager can be software and / or hardware used to manage the resources of the server cluster. Specifically, the resource manager can be a resource management platform, such as the Kubernetes platform (K8s platform). The K8s platform is a brand-new distributed architecture solution based on container technology. The K8s platform has complete cluster management capabilities, including multi-layered security protection and access control mechanisms, multi-tenant application support capabilities, transparent service registration and service discovery mechanisms, built-in intelligent load balancers, powerful fault detection and self-healing capabilities, service rolling upgrade and online expansion capabilities, scalable automatic resource scheduling mechanisms, and multi-granularity resource quota management capabilities. There are no restrictions on these aspects.

[0044] like Figure 1 As shown, the model training method includes:

[0045] S101: Obtain configuration information, where the configuration information is determined by the information of the model to be trained.

[0046] The configuration information refers to the information required for model training. This information can include resource information needed to train the model, node information used for model training, the number of server nodes required for model training parameters, the number of computing server nodes, and so on. The configuration information is used to configure the resources of the server cluster without any restrictions.

[0047] The information about the model to be trained can be related to the model, such as the number of parameters, computational cost, training task type, and sample data format. There are no restrictions on this.

[0048] In other words, the resource manager can determine the configuration information, which can be obtained by the server cluster based on the information of the model to be trained, and then trigger the subsequent flexible selection and allocation of server cluster resources based on the configuration information.

[0049] In some embodiments, the resource manager can directly receive the configuration information set by the user side with reference to the information of the model to be trained. That is, the resource manager can provide a configuration information setting interface to the user side and receive the configuration information set by the user side through the configuration information setting interface. There are no restrictions on this.

[0050] In other embodiments, the resource manager may also determine the model information to be trained, automatically configure multiple candidate configuration information based on the model information to be trained, and provide the multiple candidate configuration information to the user side for selection. The configuration information for model training is determined based on the user's selection result, without any restrictions.

[0051] S102: Determine the available resource information of the server cluster.

[0052] Among them, resources are used to describe computing resources, storage resources, distributed resources, etc. in the server cluster, while resource information can be used to describe information related to the aforementioned resources, such as the node identifier corresponding to computing resources, the storage location and storage space size corresponding to storage resources, the distributed scale and the number of migrated nodes corresponding to distributed resources, etc.

[0053] Available resource information refers to information about unused resources in the server cluster.

[0054] For example, available resource information, such as the number of nodes in a server cluster that can participate in model training tasks and the amount of sample data that can be stored, is not limited.

[0055] In this embodiment of the disclosure, an available resource information management component can be set in the resource management platform to manage and determine available resource information. Alternatively, the total resources and resources that cannot be used for model training in the server cluster can be determined, and the available resource information in the server cluster can be calculated based on the total resources and the resources that cannot be used for model training. Alternatively, any other possible implementation can be used to determine the available resource information of the server cluster, without any limitation.

[0056] In other words, the embodiments of this disclosure support the resource management platform to set up an available resource information management component, and provide a management interface for server cluster resources based on the available resource information management component. Through the management interface, the underlying resources of the server cluster are manipulated to support the elastic scaling of each server node in the server cluster during actual operation, thereby realizing the flexible allocation of resources in the server cluster.

[0057] S103: Based on the configuration information and available resource information, determine the first computing server from multiple computing servers, wherein the multiple computing servers belong to a server cluster.

[0058] The first computing server is a server in the server cluster used to execute model training tasks. There can be one or more first computing servers. When there are multiple first computing servers, each first computing server can be controlled to execute model training tasks in parallel without any restriction.

[0059] It is understood that a server cluster includes multiple servers, some of which can be used to perform computational tasks during model training. These servers can be referred to as computing servers. In this embodiment, the resource manager can select some computing servers from the multiple computing servers to perform model training tasks based on configuration information and available resource information. The computing servers selected based on configuration information and available resource information can be referred to as the first computing server.

[0060] In some embodiments, when determining the first computing server from multiple computing servers based on configuration information and available resource information, the computing power requirement of the model training task can be obtained by parsing the configuration information, and the available computing power of each of the multiple computing servers can be determined based on the available resource information. Under the premise that the sum of the available computing power of the first computing server meets the computing power requirement of the model training task, multiple computing servers can be randomly determined as the first computing server, without any restrictions.

[0061] In other embodiments, when determining the first computing server from multiple computing servers based on configuration information and available resource information, it is also possible to determine the number of computing server nodes required for the model training task based on configuration information, and select a computing server that meets the number of computing server nodes from multiple computing servers as the first computing server based on available resource information. There are no restrictions on this.

[0062] Of course, any other possible method can be used to determine the first computing server from multiple computing servers based on configuration information and available resource information. For example, the computing power of computing servers that have not performed tasks can be sorted among multiple computing servers, the required number of computing servers can be determined based on configuration information and available resource information, and computing servers can be extracted as the first computing server in order of computing power sorting. There are no restrictions on this.

[0063] S104: Generate a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained.

[0064] Among them, the task used to train the model and which can be called by the computing server can be called the model training task. The model training task is used to trigger the first computing server to train the model to be trained, thereby realizing the training of the model to be trained.

[0065] The model training task can be obtained by slicing the overall training task. For example, the model training task can correspond to an iteration stage or an iteration round of the overall training task, or a sub-task trained based on a specified training metric. There are no restrictions on this.

[0066] In this embodiment of the disclosure, when generating a model training task corresponding to the first computing server, the model performance requirements can be analyzed, and the iterative stages, iteration rounds, and required training metrics included in the model training can be determined based on the model performance requirements. Corresponding training tasks can be configured for each iterative stage, each iteration round, and each training metric to obtain the model training task. This model training task is used as the model training task corresponding to the first computing server without limitation.

[0067] In other embodiments, when generating a model training task corresponding to the first computing server, the overall training task can also be divided equally among the required amount of sample data and the number of first computing servers, so that the first computing server processes a training task with a sample data amount that conforms to the average value, thereby obtaining a model training task corresponding to the first computing server. There is no limitation on this.

[0068] Of course, model training tasks corresponding to the first computing server can also be generated in any other possible way. For example, the computing power of each first computing server can be determined, and local training tasks can be allocated based on the computing power as model training tasks corresponding to the first computing server. Alternatively, model training tasks corresponding to the first computing server can be generated based on the distributed capabilities among multiple first computing servers. There are no restrictions on this.

[0069] In this embodiment, by acquiring configuration information and determining the available resource information of the server cluster, and determining the first computing server from multiple computing servers based on the configuration information and available resource information, and generating a model training task corresponding to the first computing server, the configuration information is flexibly determined based on the information of the model to be trained, and the first computing server in the server cluster is flexibly selected and allocated based on the configuration information and the available resource information of the server cluster to execute the model training task based on the first computing server. This can effectively support the elastic scaling of nodes in the server cluster during the model training process, realize the flexible allocation of server cluster resources, effectively avoid resource waste, and improve model training efficiency.

[0070] Figure 2 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure.

[0071] like Figure 2 As shown, the model training method includes:

[0072] S201: Obtain configuration information, wherein the configuration information is determined by the information of the model to be trained, and the configuration information includes: the number of computing nodes.

[0073] In this context, a computing node is a server node used to perform corresponding computing tasks during model training. A computing node can also be called a computing server. The number of computing nodes can be used to describe the number of computing servers performing the corresponding computing tasks.

[0074] S202: Determine the available resource information for the server cluster.

[0075] For a detailed description of S202, please refer to the above embodiments, which will not be repeated here.

[0076] S203: Resource information for the computing server that determines the number of computing nodes.

[0077] The resource usage information refers to the resources used by computing servers within the server cluster that meet the required number of computing nodes. The resources used can be, for example, the total computing power resources used by computing servers with the required number of computing nodes, or the total storage resources used, etc. There are no restrictions on this.

[0078] In this embodiment of the disclosure, configuration information can be parsed to determine the number of computing nodes, select computing servers with the number of computing nodes, and determine the total computing power resources or storage resources occupied by the computing servers with the number of computing nodes as the resource occupied information, without any restrictions.

[0079] For example, if the configuration information specifies that the model training task requires 3 computing nodes, then the total computing power or storage resources occupied by the 3 computing nodes can be used as the resource information, without any restrictions.

[0080] S204: Determine the first computing server from multiple computing servers based on the occupied resource information and / or available resource information.

[0081] The resource usage information may include: first resource usage information corresponding to the number of computing servers with the first number of computing nodes, and second resource usage information corresponding to the number of computing servers with the second number of computing nodes, wherein the number of first computing nodes is less than the number of second computing nodes.

[0082] The minimum number of computing nodes that can meet the minimum training requirements of the model can be referred to as the first number of computing nodes. The resource information occupied by the computing servers with the first number of computing nodes can be referred to as the first resource information.

[0083] The maximum number of computing nodes that can meet the requirements of the model training performance index, which is preset in advance, can be called the second number of computing nodes. The resource information occupied by the computing servers with the second number of computing nodes can be called the second resource information.

[0084] In some embodiments, the resource manager can parse the setting information to determine the number of first computing nodes and the number of second computing nodes, and determine the first occupied resource information based on the number of first computing nodes and the second occupied resource information based on the number of second computing nodes, and then trigger subsequent resource allocation based on the first occupied resource information and the second occupied resource information, without any restrictions.

[0085] In this embodiment of the disclosure, the occupied resource information can be compared with the available resource information, and a strategy for selecting a first computing server from multiple computing servers can be determined based on the comparison result. This strategy can be, for example, determining the first computing server from multiple computing servers based on the occupied resource information, or determining the first computing server from multiple computing servers based on the available resource information, or determining the first computing server from multiple computing servers based on the occupied resource information combined with the available resource information, and there is no limitation on this.

[0086] Optionally, in some embodiments, if the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the first computing server is determined from multiple computing servers based on the available resource information. Since the first computing server is determined based on the available resource information when the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the computing server that meets the configuration information requirements can be selected as the first computing server, thereby ensuring the full allocation of server cluster resources and effectively improving the resource management capabilities of the resource manager.

[0087] In this embodiment of the disclosure, when the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, that is, when the available resources in the server cluster can meet the minimum requirements for model training after all model training tasks are completed and there are no extra available resources, the first computing server can be determined from multiple computing servers based on the available resource information.

[0088] In this embodiment of the present disclosure, when determining the first computing server from multiple computing servers based on available resource information, the number of third computing nodes supported by the available resource information can be determined. From the multiple computing servers, the computing server with the number of the third computing nodes is selected as the first computing server. Since the number of third computing nodes supported by the available resource information is determined and the computing server with the number of the third computing nodes is selected as the first computing server, the first computing server can be flexibly determined based on the available resource information, effectively improving the resource allocation and management capabilities of the resource manager and improving resource processing efficiency.

[0089] The number of third computing nodes refers to the number of computing servers that support model training and can fully allocate available resources. The number of third computing nodes is greater than the number of first computing nodes and less than or equal to the number of second computing nodes, and there are no restrictions on this.

[0090] In this embodiment of the disclosure, the number of computing servers in the server cluster that can participate in the model training task can be determined based on the available resource information, and this number is used as the number of third computing nodes, without any limitation.

[0091] In this embodiment of the disclosure, since the available resource information in the server cluster meets the requirements of model training, the model training task can be executed based on the available resource information of the computing server with the third number of computing nodes.

[0092] Optionally, in other embodiments, if the available resource information is greater than the second occupied resource information, the first computing server is determined from multiple computing servers based on the second occupied resource information. Since the first computing server is determined based on the second occupied resource information when the available resource information is greater than the second occupied resource information, the computing server that meets the configuration information requirements can be selected as the first computing server, thereby ensuring the full allocation of server cluster resources and effectively improving the resource management capabilities of the resource manager.

[0093] In this embodiment of the disclosure, when the available resource information is greater than the second occupied resource information, that is, when the available resources in the server cluster are more than the resource requirements for model training, the resources corresponding to the second occupied resource information can be used as the resources for model training, and the first computing server can be determined from multiple computing servers based on the second occupied resource information.

[0094] In this embodiment of the disclosure, when determining the first computing server from multiple computing servers based on the second resource occupancy information, a computing server with a second number of computing nodes can be selected as the first computing server from the multiple computing servers. Since a computing server with a second number of computing nodes is selected as the first computing server, the first computing server that meets the configuration information requirements can be flexibly selected, effectively improving the resource management capabilities of the resource manager, effectively avoiding resource waste caused by excessive resource allocation, and improving the effective utilization rate of server cluster resources.

[0095] In this embodiment of the disclosure, the number of second nodes required for configuration information can be determined based on the second resource occupancy information, and the computing server with the number of second computing nodes can be used as the first computing server.

[0096] S205: If the available resource information is less than or equal to the first occupied resource information, then determine the current resource information of the server cluster.

[0097] The current resource information refers to the resource information that the server cluster can currently use for model training when the available resource information is less than or equal to the first occupied resource information.

[0098] In this embodiment of the disclosure, when the available resource information is less than or equal to the first occupied resource information, that is, when the resources in the server cluster that can perform model training tasks are less than the minimum requirement of the configuration information, the resource information that can perform model training in the current server cluster can be determined as the current resource information.

[0099] S206: Update the available resource information based on the current resource information.

[0100] In this embodiment of the disclosure, a current resource information detection component can be set in the resource management platform to continuously detect the current resource information and update the available resource information in real time. Alternatively, an available resource information update cycle can be set to periodically detect the current resource information and update the available resource information after the current resource information changes. There are no restrictions on this.

[0101] In this embodiment of the disclosure, when it is detected that the current resource information is greater than the first occupied resource information, the available resource information is updated, and the first computing server is determined from multiple computing servers based on the occupied resource information and / or the available resource information.

[0102] In other embodiments, the current resource information can be updated to available resource information, and the difference between the first occupied resource information and the available resource information can be sent to the user side based on the current resource information to request the user side to allocate server cluster resources according to the difference between the first occupied resource information and the available resource information. There are no restrictions on this.

[0103] S207: Generate a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained.

[0104] For a detailed description of S207, please refer to the above embodiments, which will not be repeated here.

[0105] In this embodiment, since the configuration information is flexibly determined based on the information of the model to be trained, and the first computing server in the server cluster is flexibly selected and allocated based on the configuration information and the available resource information of the server cluster, so as to execute the model training task based on the first computing server, the elastic scaling of nodes in the server cluster can be effectively supported during the model training process. This enables flexible allocation of server cluster resources, effectively avoiding resource waste and improving model training efficiency. Since the resource information occupied by the computing server is determined based on the number of computing nodes, and the first computing server is determined from multiple computing servers based on the occupied resource information and / or available resource information, the first computing server can be flexibly selected and flexibly controlled to participate in model training. This allows for flexible scheduling of resources in the server cluster based on the occupied resource information and available resource information. Furthermore, since the first computing server is determined based on the available resource information when the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the available resources... When the source information is greater than the second occupied resource information, the first computing server is determined based on the second occupied resource information. This allows for the selection of a computing server that meets the configuration information requirements as the first computing server, ensuring sufficient allocation of server cluster resources and effectively improving the resource management capabilities of the resource manager. Since the available resource information is updated based on the current resource information when the available resource information is less than or equal to the first occupied resource information, it can update the available resource information based on the current resource information when the available resource information does not meet the configuration information requirements, until the available resource information meets the configuration information requirements. This ensures that the first computing server is automatically determined when the available resource information meets the configuration information requirements, improving the stability of model training while enhancing the automation of model training. Since the computing server with the number of third computing nodes is selected as the first computing server, it can flexibly determine the first computing server based on the available resource information, ensuring sufficient allocation of server cluster resources, effectively improving resource management capabilities, and reducing resource waste caused by excessive resource allocation.

[0106] Figure 3 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure.

[0107] like Figure 3 As shown, the model training method includes:

[0108] S301: Obtain configuration information, wherein the configuration information is determined by the information of the model to be trained.

[0109] S302: Determine the available resource information for the server cluster.

[0110] S303: Determine the resource information occupied by the computing server with the number of computing nodes, wherein the resource information includes: first resource information corresponding to the computing server with the first number of computing nodes, and second resource information corresponding to the computing server with the second number of computing nodes, wherein the first number of computing nodes is less than the second number of computing nodes.

[0111] For a detailed description of S301-S303, please refer to the above embodiments, which will not be repeated here.

[0112] S304: If the available resource information is greater than the first occupied resource information, then select a computing server with the first number of computing nodes from multiple computing servers as the first computing server.

[0113] In this embodiment of the disclosure, if the available resource information is greater than the first occupied resource information, that is, if the resources in the server cluster that can perform model training tasks are greater than the preset minimum resource requirements, the computing server with the first number of computing nodes can be directly selected as the first computing server.

[0114] In other words, if the available resources in the server cluster meet the minimum resource requirements for model training, the portion of computing servers with the fewest computing nodes required for model training can be selected as the first computing servers, and the startup of these computing servers can be triggered to execute the model training task. Subsequently, the startup of model training tasks on each subsequent computing server can be triggered. During the model training process, when the available resource information of the server cluster monitored by the resource manager is sufficient, the resource manager can start other computing servers one after another. The started computing servers will join the model training task in turn. In this embodiment, the total number of computing nodes of the newly started computing servers does not exceed the maximum number of computing nodes described based on the configuration information, effectively avoiding the waste of server cluster resources and avoiding affecting the effective execution of other model training tasks.

[0115] S305: Select a computing server with a fourth number of computing nodes from the remaining computing servers as the first computing server, wherein the sum of the number of the first computing nodes and the number of the fourth computing nodes is less than or equal to the number of the second computing nodes, and the number of the fourth computing nodes is determined by the current resource information of the server cluster.

[0116] The remaining computing servers are the computing servers remaining in the server cluster after removing the first computing server. After selecting a number of computing nodes from multiple computing servers as the first computing server, the remaining computing servers in the server cluster can be called the remaining computing servers.

[0117] The fourth number of computing nodes is used to regulate and supplement the model training. The sum of the fourth number of computing nodes and the first number of computing nodes is less than or equal to the second number of computing nodes. This is to ensure that when using computing servers with the first number of computing nodes for model training, even after all computing servers with the fourth number of computing nodes are used for model training, the configuration information requirements can still be met without excessively occupying the resources of the server cluster.

[0118] In this embodiment of the disclosure, when it is determined that the available resource information is greater than the first occupied resource information, a computing server with the first number of computing nodes can be selected as the first computing server first. Then, the available resource information is compared with the second occupied resource information, and the fourth number of computing nodes is determined according to the comparison result. Computing servers with the fourth number of computing nodes are successively selected from the remaining computing servers as the first computing server. There is no limitation on this.

[0119] In this embodiment of the disclosure, the resources corresponding to the first resource occupancy information can be allocated to computing servers with a first number of computing nodes. The computing servers with a first number of computing nodes are used for model training. That is, the model training task is started first with computing servers with the minimum number of nodes. Then, it is checked whether there are any extra available resources in the server. If there are, a computing server with a fourth number of computing nodes is determined from the remaining computing servers, and the computing server with the fourth number of computing nodes is added as the first computing server to the model training task, so as to realize the elastic expansion of resources when running the model training task. There are no restrictions on this.

[0120] S306: Generate a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained.

[0121] For a detailed description of S306, please refer to the above embodiments, which will not be repeated here.

[0122] In this embodiment of the disclosure, such as Figure 4 As shown, Figure 4This is a schematic diagram of model training control proposed in another embodiment of this disclosure. The resource manager receives configuration information submitted by the user. The configuration information includes the number of first computing nodes and the number of second computing nodes. It checks whether the number of computing nodes in the available resource information of the server cluster is greater than the number of first computing nodes. If it is greater than the number of first computing nodes, it checks whether the number of computing nodes in the available resource information of the server cluster is greater than the number of second computing nodes. If it is greater than the number of second computing nodes, it selects a computing server with the number of second computing nodes as the first computing server based on the second occupied resource information. If it is not greater than the number of second computing nodes, it selects a computing server with the number of third computing nodes as the first computing server based on the available resource information. If it is not greater than the number of first computing nodes, it can wait for available resources until the number of computing nodes in the available resource information of the server cluster is greater than the number of first computing nodes. In this process, a resource monitoring component can also be set up. When available resources are found, the first computing server to be added is determined based on the available resources. If an abnormal situation occurs that causes a shortage of resources in use, some first computing servers can be randomly stopped according to the resource gap, while ensuring that the number of computing nodes performing model training tasks is not less than the number of first computing nodes.

[0123] In this embodiment, the configuration information is flexibly determined based on the information of the model to be trained, and the first computing server in the server cluster is flexibly selected and allocated based on this configuration information and the available resource information of the server cluster. The model training task is executed on the first computing server, which effectively supports the elastic scaling of nodes in the server cluster during model training. This enables flexible allocation of server cluster resources, effectively avoiding resource waste and improving model training efficiency. Since the first computing server is selected from multiple computing servers when the available resource information is greater than the first occupied resource information, the model training can be started directly with the minimum number of computing nodes. Furthermore, it supports dynamically adding the first computing server during model training, enabling rapid resource acquisition to start model training and improving model training efficiency. It allows for elastic scaling of the computing servers participating in model training, improving model training efficiency and enhancing the flexibility and convenience of resource scheduling. Moreover, the total number of computing nodes on the newly started computing servers does not exceed the maximum number of computing nodes described based on the configuration information, effectively avoiding resource waste in the server cluster and preventing interference with the effective execution of other model training tasks.

[0124] Figure 5 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure.

[0125] like Figure 5 As shown, the model training method includes:

[0126] S501: Obtain configuration information, which is determined by the information of the model to be trained. The configuration information includes: the number of parameter nodes and domain name configuration information.

[0127] The number of parameter nodes refers to the number of parameter servers, which can be used to determine the number of parameter servers required for the model training task.

[0128] Among them, the parameter server is a type of server in the server cluster. The parameter server can be used to store the parameters in the model to be trained, and iteratively update the parameters in the model to be trained according to the calculation of the computing server, so as to realize the training of the model.

[0129] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram of a model training architecture proposed in another embodiment of the present disclosure. The computing node (i.e., the computing server) receives the corresponding data fragments of the model to be trained and performs training. It periodically pushes gradients to the parameter server and pulls the latest model parameters. The parameter server is used to divide and store model parameters and is responsible for updating the parameters. The computing node (i.e., the computing server) and the parameter server are combined to form the model training architecture.

[0130] The information used to configure the domain names of each node (computation server or parameter server) in the model training task can be called domain name configuration information. This domain name configuration information can be used to enable data communication between different nodes (computation server or parameter server) without any restrictions.

[0131] In this embodiment of the disclosure, the sum of the number of parameter nodes and the number of computing nodes in the model training task can be determined through configuration information, and a corresponding number of domain name configuration information can be set. Alternatively, the number of all servers in the server cluster can be determined directly, and a corresponding number of domain name configuration information can be set. There is no limitation on this.

[0132] In this embodiment of the disclosure, the domain name configuration information of the model training task can also be set in a gradient manner, that is, the domain name configuration information of the overall model training task can be configured. The multiple model training tasks are divided into sub-tasks of the overall training task, and the domain name configuration information of the sub-tasks is configured. By using gradient configuration, the model training task can be made more organized, thereby ensuring the orderly operation of each sub-task.

[0133] S502: Determine the available resource information for the server cluster.

[0134] S503: Based on configuration information and available resource information, determine the first computing server from multiple computing servers, where the multiple computing servers belong to a server cluster.

[0135] For a detailed description of S501-S503, please refer to the above embodiments, which will not be repeated here.

[0136] S504: Parameter identifier that determines the number of parameter nodes.

[0137] Among them, the parameter identifier is the identification information corresponding to the parameter server. The parameter identifier can be used to uniquely identify the corresponding parameter server. The parameter identifier can be a combination of numbers, letters, symbols, etc., so as to more conveniently identify the parameter server. There are no restrictions on this.

[0138] In this embodiment of the disclosure, corresponding parameter identifiers can be pre-configured for all parameter servers in the server cluster, and then the parameter identifiers for the number of parameter nodes used for model training can be determined from them. Alternatively, the number of parameter nodes can be determined, and then the corresponding number of parameter identifiers can be configured according to the number of parameter nodes. Or, any other possible implementation method can be used to determine the parameter identifiers for the number of parameter nodes, such as setting up a parameter server management component in the resource management platform and configuring the parameter identifiers for the number of parameter nodes based on the parameter server management component. There are no restrictions on this.

[0139] S505: Determine the first computing identifier of the first computing server.

[0140] The identification information used to identify the first computing server can be called the first computing identifier. The first computing identifier can be a combination of numbers, letters, symbols, etc., to uniquely identify the corresponding first computing server based on the first computing identifier, without any restrictions.

[0141] In this embodiment of the disclosure, after determining the number of computing nodes in the first computing server, the corresponding first computing identifier can be configured according to the number of computing nodes. Alternatively, all computing servers in the server cluster can be identified in advance, and then the first computing identifier of the first computing server can be determined according to the selection of the first computing server. There are no restrictions on this.

[0142] S506: Configure the parameter server to which the parameter belongs based on the domain name configuration information. The parameter server belongs to the server cluster.

[0143] In this embodiment of the disclosure, parameter servers to which parameter identifiers belong can be configured according to the domain name configuration information, and the parameter servers can be bound to the domain names described in the corresponding domain name configuration information, so that the resource manager can train the relevant parameter servers according to the domain name search model, and enable data communication between different nodes (computing servers or parameter servers), without any limitations.

[0144] In this embodiment of the disclosure, the process of configuring the parameter server to which the parameter identifier belongs based on the domain name configuration information may involve combining the parameter identifier with the domain name and using the combined identifier information as the identifier of the corresponding parameter server. Alternatively, a domain name and parameter identifier matching table may be set up, and the domain name may be configured to the corresponding parameter identifier in the table. Or, any other possible implementation method may be used to configure the parameter server to which the parameter identifier belongs based on the domain name configuration information, without any limitation.

[0145] For example, when the model training task starts, the domain name configuration information can be arranged. When the parameter servers are started one after another, each time a parameter server is started, a domain name configuration information is bound to the parameter identifier of that parameter server in the order of arrangement, until all parameter servers are started and the domain name configuration information is confirmed to be configured.

[0146] S507: Configure the first computing server according to the domain name configuration information.

[0147] In this embodiment of the disclosure, the first computing server can be bound to the domain name corresponding to the domain name configuration information so that the resource manager can query the first computing server according to the domain name configuration information, and there are no restrictions on this.

[0148] In this embodiment of the disclosure, the domain name configuration information can be directly combined with the first computing identifier, and the combined identifier information can be used as the identifier of the computing server. Alternatively, code or data for identifying the domain name configuration can be added to the first computing identifier. Or, any other possible implementation method can be used to configure the first computing server according to the domain name configuration information, without any limitation.

[0149] For example, when the model training task starts, the domain name configuration information can be arranged. When the first computing server is started one after another, each time the first computing server is started, a domain name configuration information is bound to the first computing identifier of the first computing server in the order of arrangement, until all the first computing servers are started and the domain name configuration information is confirmed to be configured.

[0150] S508: Control the first computing server to train the model to be trained based on the model training task, wherein the first computing server obtains model parameters from the parameter server to train the model to be trained.

[0151] In this embodiment of the disclosure, multiple first computing servers can be selected and controlled in parallel according to the domain name configuration information to train the model to be trained based on the model training task. The first computing server obtains model parameters from the parameter server. The parameter server can continuously update the model parameters according to the calculation results of the first computing server and return the latest model parameters to the first computing server. There are no restrictions on this.

[0152] Optionally, in this embodiment of the present disclosure, controlling the first computing server to train the model to be trained based on the model training task may involve pulling the training task image file corresponding to the model training task, running the training task image file, creating a container environment corresponding to the model training task, wherein the container environment is used to configure the environment information for running the model training task, and sending a task start message to the first computing server, wherein the task start message includes a task identifier. In response to the task start message, the first computing server starts the training process corresponding to the task identifier to train the model to be trained. Since the container environment is created based on pulling the training task image file to start the model training task, the model training task can be dynamically managed using the container environment, which facilitates the dynamic scaling up and down of the process nodes of the model training task in the container environment, thereby improving the flexibility of model training task management.

[0153] In this embodiment of the disclosure, the resource manager can pre-configure the training task image file corresponding to the model training task and store it in the relevant storage device. When the model training task is executed, the training task image file is pulled from the relevant storage device, and a container environment is created according to the relevant data information in the training task image file. The container environment is the environment information in which the model training task is trained. The container environment can be, for example, an image environment or a virtual machine environment. The environment information can be, for example, a language environment or a software environment, and there are no restrictions on this.

[0154] The resource manager can be, for example, a resource management platform like Kubernetes. It can then manage server cluster resources and run training task image files in a container environment based on the Kubernetes platform, without any restrictions.

[0155] In this embodiment of the disclosure, the process of running a training task image file in a container environment may involve generating a model training task in the container environment based on the task image file, and then starting the first computing server to execute the model training task based on the resources allocated by the resource management platform to the first computing server.

[0156] In this embodiment of the disclosure, when the first computing server is started to execute the model training task, a container environment can be built according to the training task image file, the parameter data related to the model training task can be extracted, the training task image file can be run in the container environment using a resource management platform, and the parameters can be updated according to the running results to realize model training.

[0157] S509: If it is determined that resource allocation is to be carried out, then determine the priority of the model training task.

[0158] This allows you to configure the priority of each of the multiple running tasks so that resources can be allocated according to priority.

[0159] In this embodiment of the disclosure, when resources are scarce or when a resource allocation instruction is received from the user side, it can be determined that resources in the server cluster need to be allocated. At this time, the priority of the model training task can be queried, and the resource allocation situation can be determined according to the priority of the model training task.

[0160] In this embodiment of the disclosure, priority can be represented by numbers, such as priority level 1, level 2, etc. The larger the value, the higher the priority. Alternatively, Chinese characters, letters, symbols, etc., can be used to represent the degree of priority, such as dividing the priority into "low", "medium", "high", etc. There are no restrictions on this.

[0161] S510: If the priority is less than or equal to the priority threshold, control the first computing server to stop model training based on the model training task.

[0162] The priority threshold is a pre-set priority threshold value used to manage the start and stop of the first computing server.

[0163] In this embodiment of the disclosure, a priority threshold for model training tasks can be set on the resource management side. When the priority of a model training task is less than or equal to the priority threshold, model training based on the model training task is stopped, and the resources of the computing server occupied by the model training task are released to realize resource allocation. For example, the released resources can be allocated to tasks with higher priority, and there is no restriction on this.

[0164] For example, a priority threshold can be set to "medium". When the server cluster experiences resource shortage and the priority of the model training task is determined to be "low", the first computing server for some model training can be shut down to free up resources for higher priority tasks.

[0165] Optionally, in this embodiment of the present disclosure, controlling the first computing server to stop model training based on the model training task can be done by shutting down the container environment corresponding to the model training task or by sending a process stop message to the first computing server. In response to the process stop message, the first computing server stops the training process corresponding to the task identifier. Since the training process corresponding to the task identifier is stopped by shutting down the container environment corresponding to the model training task or by sending a process stop message, the method of stopping the training process can be flexibly selected. Using the process stop message, the operation of the first computing server corresponding to the task identifier can be flexibly stopped, thereby releasing resources and enabling the resource manager to conveniently control the underlying resources in the server cluster, realizing the elastic scaling of the computing server during training.

[0166] In this embodiment of the disclosure, the process stop message can be used to stop the training process running on the computing server. The process stop message can specify one or more computing servers to stop the training process, or it can directly delete the container environment to stop the model training process, or it can use any other possible method to stop the model training process, such as disabling the model training task for a period of time until a higher priority task is completed and then restarting it, etc. There are no restrictions on this.

[0167] In this embodiment of the disclosure, when the priority of the model training task is greater than the priority threshold, the resource allocation request can be ignored and the model training task can continue to run.

[0168] In this embodiment of the disclosure, the priorities of multiple tasks can also be compared, and resources can be allocated in descending order of priority.

[0169] For example, when platform resources are scarce, if a high-priority task requires resources to start, the platform will select a certain number of computing servers from low-priority tasks for recycling, in order to free up resources for the operation of high-priority tasks.

[0170] In this embodiment, by flexibly determining the configuration information based on the information of the model to be trained, and flexibly selecting and allocating the first computing server in the server cluster based on this configuration information and the available resource information of the server cluster, the model training task can be executed based on the first computing server. This effectively supports the elastic scaling of nodes in the server cluster during the model training process, enabling flexible allocation of server cluster resources, effectively avoiding resource waste, and improving model training efficiency. Since the parameter identifier belongs to the parameter server based on the domain name configuration information, and the first computing server is configured based on the domain name configuration information, the domain name can be used to bind the parameter server and the first computing server. This ensures that the first computing server can only see the parameter server and is not affected by other computing servers, thereby enhancing the fault tolerance of model training and improving the anomaly handling capability of model training. Since the first computing server is controlled to train the model to be trained based on the model training task, the training efficiency of the model training task can be effectively improved, enhancing the flexibility and reliability of model training. Because it creates a container environment based on the generated training task image file, and runs the training task image file within the container environment to start the model training task, it allows for dynamic management of the model training task. This facilitates dynamic scaling up and down of the process nodes of the model training task within the container environment, improving the flexibility of model training task management. Since it assigns priorities to model training tasks, it allows for flexible resource allocation based on priorities, ensuring effective utilization of server cluster resources in multi-tasking scenarios and improving task execution performance. Because it stops the training process corresponding to the task identifier by closing the container environment corresponding to the model training task or by sending a process stop message, it allows for flexible selection of the method of stopping the training process. Using process stop messages, it is possible to flexibly stop the operation of the first computing server corresponding to the task identifier, thereby releasing resources. This allows resource managers to easily control the underlying resources in the server cluster, achieving elastic scaling of computing servers during training.

[0171] Figure 7 This is a schematic flowchart of a model training method proposed in one embodiment of this disclosure.

[0172] like Figure 7 As shown, this model training method is applied to a server cluster, which includes multiple computing servers, including:

[0173] S701: Provide the resource manager with information about the model to be trained, wherein the information about the model to be trained is used to determine the configuration information.

[0174] In this embodiment of the disclosure, the server cluster can organize the information of the model to be trained and provide it to the resource manager so that the resource manager can determine the configuration information based on the information of the model to be trained.

[0175] In this embodiment of the disclosure, the resource information required to run the model training task can also be calculated in the server cluster according to the model to be trained, and the resource information can be sent to the resource manager as the model to be trained information, which can be determined or adjusted by the resource manager as configuration information, and there are no restrictions on this.

[0176] S702: Provides the resource manager with information on the available resources of the server cluster.

[0177] In this embodiment of the disclosure, an available resource information transmission interface can be set to provide available resource information to the resource manager in real time, or an available resource information transmission period can be set to periodically provide available resource information of the server cluster to the resource manager.

[0178] In other embodiments, an available resource information monitoring component can be set up in the server cluster to monitor the available resource information in the server cluster in real time. When the resource manager needs available resource information, it sends an available resource information confirmation request to the server cluster. The server cluster receives and responds to the available resource information confirmation request and provides the available resource information of the server cluster to the resource manager. There are no restrictions on this.

[0179] S703: Based on the model training task generated by the resource manager, determine the first computing server, wherein the first computing server is determined by the resource manager from multiple computing servers based on configuration information and available resource information.

[0180] In this embodiment of the disclosure, in the server cluster, the first computing server is determined based on the model training task generated by the resource manager. This can be done by the resource manager based on the first and second occupied resource information in the configuration information, and then combined with the available resource information of the server cluster. There are no restrictions on this method.

[0181] S704: Controls the first computing server to train the model to be trained based on the model training task.

[0182] In this embodiment of the disclosure, in the step of controlling the first computing server to train the model to be trained based on the model training task, a mirror training task can be set up, and the first computing server can be controlled to run the mirror training task in the container environment to train the model to be trained.

[0183] In this embodiment of the disclosure, when there are multiple first computing servers, multiple first computing servers can be used to run in parallel to train the model to be trained, and there is no limitation on this.

[0184] In this embodiment, by providing the resource manager with information on the model to be trained and the available resources of the server cluster, and based on the model training task generated by the resource manager, a first computing server is determined and controlled. The model to be trained is then trained based on the model training task. Since the resource manager is provided with information on the model to be trained and the available resources of the server cluster, the resource manager can flexibly select and allocate the first computing server in the server cluster based on this configuration information and the available resources of the server cluster. When the model training task is executed based on the first computing server, the elastic scaling of nodes in the server cluster can be effectively supported during the model training process, enabling flexible allocation of server cluster resources, effectively avoiding resource waste, and improving model training efficiency.

[0185] Figure 8 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure.

[0186] like Figure 8 As shown, the model training method includes:

[0187] S801: Provide the resource manager with information about the model to be trained, wherein the information about the model to be trained is used to determine the configuration information.

[0188] S802: Provides the resource manager with information on the available resources of the server cluster.

[0189] S803: Based on the model training task generated by the resource manager, determine the first computing server, wherein the first computing server is determined by the resource manager from multiple computing servers based on configuration information and available resource information.

[0190] For a detailed description of S801-S803, please refer to the above embodiments, which will not be repeated here.

[0191] S804: Receive a task start message sent by the resource manager, wherein the task start message includes: task identifier.

[0192] The data information used to initiate the model training task can be called the task initiation message. The task initiation message contains a task identifier to identify the model training task that needs to be initiated.

[0193] In this embodiment of the disclosure, after generating a model training task, the resource manager can send a task start message to the server cluster based on the currently available resource information. The server cluster receives the task start message in real time to trigger the first computing server to execute the model training task corresponding to the task identifier.

[0194] In this embodiment of the disclosure, an interface for task initiation can be set up to connect the resource manager to the server cluster. The server cluster uses this interface to receive the task initiation message sent by the resource manager. Alternatively, a wireless transmission method can be used for the server cluster to receive the task initiation message sent by the resource manager. Or, any other possible implementation method can be used to receive the task initiation message sent by the resource manager. There are no restrictions on this.

[0195] S805: Start the training process corresponding to the task identifier in the first computing server to train the model to be trained.

[0196] In this embodiment of the disclosure, the first computing server starts the training process corresponding to the task identifier, which may be done after determining the task start message and in response to the task start message, starting the training process corresponding to the task identifier.

[0197] In other words, the task start message can start the training process corresponding to the task identifier. In an overall training task, there can be multiple subtasks. The task identifier can uniquely identify the corresponding subtask, so as to realize the flexible control of the start of multiple tasks based on the task start message.

[0198] In this embodiment of the disclosure, there can be multiple first computing servers. Therefore, different training processes can be configured for different first computing servers and distinguished according to the task identifier. The task identifier is added to the task start message so that the training process corresponding to the task identifier can be started accurately to train the model to be trained. There are no restrictions on this.

[0199] S806: Receives process stop messages sent by the resource manager.

[0200] In this embodiment of the disclosure, the process stop message can share the same interface as the task start message to facilitate the start and stop of the server cluster management model training task. Alternatively, the process stop message sent by the resource manager can be received via wireless transmission. There are no restrictions on this.

[0201] S807: In response to a process stop message, stop the training process corresponding to the task identifier.

[0202] In this embodiment of the disclosure, different task identifiers can uniquely identify a training process executed on a first computing server. Therefore, a task identifier that needs to be stopped can be added to the process stop message. After receiving the process stop message, the server cluster can parse the task identifier in it and determine to stop the training process corresponding to the task identifier. This enables flexible control of the task process and effectively improves the controllable scaling of model training.

[0203] In this embodiment, by providing the resource manager with information about the model to be trained and the available resources of the server cluster, the resource manager can flexibly select and allocate the first computing server in the server cluster based on this configuration information and available resource information. When the model training task is executed based on the first computing server, it can effectively support the elastic scaling of nodes in the server cluster during the model training process, realize flexible allocation of server cluster resources, effectively avoid resource waste, and improve model training efficiency. Since the corresponding training process is stopped by setting a process stop message, the corresponding task process can be stopped according to the process stop message. During the execution of the model training task, the first computing server in the server cluster can be dynamically and flexibly managed, thereby allowing the first computing server to release resources and ensuring the flexibility of resource management.

[0204] Figure 9 This is a schematic flowchart of a model training method proposed in another embodiment of this disclosure.

[0205] like Figure 9 As shown, the model training method includes:

[0206] S901: Provide the resource manager with information about the model to be trained, wherein the information about the model to be trained is used to determine the configuration information.

[0207] S902: Provides the resource manager with information on the available resources of the server cluster.

[0208] S903: Based on the model training task generated by the resource manager, determine the first computing server, wherein the first computing server is determined by the resource manager from multiple computing servers based on configuration information and available resource information.

[0209] For a detailed description of S901-S903, please refer to the above embodiments, which will not be repeated here.

[0210] S904: Controls the first computing server to obtain model parameters from the parameter server.

[0211] In this embodiment of the disclosure, since the model parameters will be updated according to the model training, the model parameter acquisition period can be set to periodically update the model parameters in the first computing server. Alternatively, the updated model parameters can be obtained from the parameter server after the first computing server completes a certain training task. There is no limitation on this.

[0212] S905: Controls the first computing server to obtain training sample data based on the data distribution server.

[0213] The server cluster also includes a data distribution server, which reads training sample data from local or remote storage and initiates a data distribution service to distribute the training sample data to multiple first computing servers.

[0214] Training sample data refers to the data information used as samples for model training. The training sample data can be used as samples and combined with model parameters by a computing server to train the model to be trained.

[0215] In this embodiment of the disclosure, the data distribution server can divide the training sample data into multiple corresponding data fragments and distribute them to different first computing servers. Alternatively, the training sample data can be pre-distributed in local or remote storage, stored in the form of data fragments, and then distributed to different first computing servers by the data distribution server. There is no limitation on this.

[0216] Optionally, in this embodiment of the present disclosure, the data distribution server includes: a data distribution service and a data snapshot service. The data reading location information is determined based on the data snapshot service and provided to the data distribution service. Sample data corresponding to the data reading location information is obtained as training sample data based on the data distribution service. Since the data reading location information is determined based on the data snapshot service and the sample data corresponding to the data reading location information is obtained as training sample data based on the data distribution service, the training sample data can be flexibly allocated based on the data distribution service, improving the flexibility of model training. The stability of model training is guaranteed based on the data snapshot service, effectively improving the fault tolerance of model training and improving the efficiency of model training.

[0217] The service used to record the data reading location information of training sample data in multiple first computing servers can be called the data snapshot service. When a first computing server experiences an anomaly or restarts suddenly, the data snapshot service can be used to redirect the training sample data at the data reading location and continue model training.

[0218] Among them, the data reading location information is the storage location information corresponding to the training sample data. The training sample data is divided into multiple data slices, and different data slices have corresponding data reading location information. Alternatively, when the first computing server runs the model training task, the data points corresponding to the training sample data can be recorded in real time and the data points can be used as the data reading location information. There are no restrictions on this.

[0219] In this embodiment of the disclosure, when an abnormal situation causes the first computing server to restart, or when a new first computing server participates in the model training task, the data reading location information can be determined according to the data snapshot service, and the data reading location information can be provided to the data distribution service. The data distribution service distributes the training sample data corresponding to the data reading location information to the restarted first computing server or the new first computing server, so that the model training task can continue based on the data reading location. No restrictions are imposed on this.

[0220] For example, when a training anomaly occurs on the first computing server, the data snapshot service can record the data breakpoint where the anomaly occurred in real time. When the first computing server restarts, the training sample data can be reread from the data breakpoint location.

[0221] In this embodiment of the disclosure, such as Figure 10 As shown, Figure 10 This is a schematic diagram of a data distribution server proposed in another embodiment of the present disclosure. The data distribution server includes a data reading service, a data distribution service, and a data snapshot service. The data reading service is used to read training sample data from local or remote storage. The data distribution service is used to distribute the training sample data to multiple first computing servers. The data snapshot service is used to record the data reading location information of the training sample data in multiple first computing servers in real time. Thus, the training sample data can be flexibly allocated.

[0222] S906: Controls the first computing server to train the model to be trained based on model parameters and training sample data.

[0223] In this embodiment of the disclosure, a mirrored model to be trained can be built in the first computing server based on model parameters, and training sample data can be input into the mirrored model to be trained. The model parameters can be updated using gradient training or iterative training to train the model to be trained.

[0224] In this embodiment, by providing the resource manager with information about the model to be trained and the available resources of the server cluster, the resource manager can flexibly select and allocate the first computing server in the server cluster based on this configuration information and available resources. When the model training task is executed based on the first computing server, the elastic scaling of nodes in the server cluster can be effectively supported during the model training process, enabling flexible allocation of server cluster resources, effectively avoiding resource waste, and improving model training efficiency. Since the first computing server is controlled to train the model to be trained based on model parameters and training sample data, the accuracy and stability of model training can be effectively improved, thus increasing model training efficiency. Because the data reading location information is determined based on the data snapshot service, and the sample data corresponding to the data reading location information is obtained as training sample data based on the data distribution service, the training sample data can be flexibly allocated based on the data distribution service, improving the flexibility of model training. The stability of model training is ensured based on the data snapshot service, effectively improving the fault tolerance of model training and increasing model training efficiency.

[0225] In summary, such as Figure 11 As shown, Figure 11 This is a schematic diagram of a model training scheme proposed in another embodiment of this disclosure. The server cluster includes a parameter server, a first computing server, and a data distribution server. The data distribution server flexibly manages the task process. The resource management side includes a resource pool, and the platform can flexibly start and stop the first computing servers to achieve effective management of server cluster resources. In the model training scheme of this embodiment, each first computing server can only see its corresponding parameter server and cannot see other first computing servers, thus achieving no awareness of anomalies in other first computing servers. This effectively improves the fault tolerance of model training. Furthermore, when the model training task is initially started, it is not necessary to wait for all first computing servers to start before starting the entire model training task; the first computing server that starts first can begin training first, thereby improving model training efficiency.

[0226] Figure 12 This is a schematic diagram of the structure of a model training device proposed in one embodiment of this disclosure.

[0227] like Figure 12 As shown, the model training device 120 is used by a resource manager who manages the resources of the server cluster, including:

[0228] The acquisition module 1201 is used to acquire configuration information, wherein the configuration information is determined by the information of the model to be trained;

[0229] The first determining module 1202 is used to determine the available resource information of the server cluster;

[0230] The second determining module 1203 is used to determine a first computing server from multiple computing servers based on configuration information and available resource information, wherein the multiple computing servers belong to a server cluster; and

[0231] The generation module 1204 is used to generate a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained.

[0232] In some embodiments of this disclosure, such as Figure 13 As shown, Figure 13 This is a schematic diagram of a model training apparatus according to another embodiment of this disclosure. Configuration information includes: the number of computing nodes; wherein, the second determining module 1203 includes:

[0233] The first determining submodule 12031 is used to determine the resource information occupied by the computing server with the number of computing nodes;

[0234] The second determining submodule 12032 is used to determine the first computing server from multiple computing servers based on the occupied resource information and / or available resource information.

[0235] In some embodiments of this disclosure, such as Figure 13 As shown, the resource information occupied includes: first resource information corresponding to the number of computing servers with the first number of computing nodes, and second resource information corresponding to the number of computing servers with the second number of computing nodes, wherein the number of first computing nodes is less than the number of second computing nodes.

[0236] In some embodiments of this disclosure, such as Figure 13 As shown, the second determining submodule 12032 is specifically used for:

[0237] When the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the first computing server is determined from multiple computing servers based on the available resource information.

[0238] When the available resource information is greater than the second occupied resource information, the first computing server is determined from multiple computing servers based on the second occupied resource information.

[0239] In some embodiments of this disclosure, such as Figure 13 As shown, it also includes:

[0240] The third determining module 1205 is used to determine the current resource information of the server cluster when the available resource information is less than or equal to the first occupied resource information;

[0241] The update module 1206 is used to update the available resource information based on the current resource information.

[0242] In some embodiments of this disclosure, such as Figure 13 As shown, the second determining submodule 12032 is specifically used for:

[0243] Determine the number of third computing nodes supported by available resource information;

[0244] From multiple computing servers, a computing server with a third number of computing nodes is selected as the first computing server.

[0245] In some embodiments of this disclosure, such as Figure 13 As shown, the second determining submodule 12032 is specifically used for:

[0246] From multiple computing servers, select a computing server with a second number of computing nodes as the first computing server.

[0247] In some embodiments of this disclosure, such as Figure 13 As shown, the configuration information also includes: the number of parameter nodes and domain name configuration information; among which, it also includes:

[0248] The fourth determining module 1207 is used to determine the parameter identifier of the number of parameter nodes before generating the model training task corresponding to the first computing server;

[0249] The fifth determining module 1208 is used to determine the first computing identifier of the first computing server;

[0250] The first configuration module 1209 is used to configure the parameter server to which the parameter identifier belongs based on the domain name configuration information, wherein the parameter server belongs to the server cluster;

[0251] The second configuration module 1210 is used to configure the first computing server according to the domain name configuration information.

[0252] In some embodiments of this disclosure, such as Figure 13 As shown, the generation module 1204 is specifically used for:

[0253] The first computing server is controlled to train the model to be trained based on the model training task. The first computing server obtains model parameters from the parameter server to train the model to be trained.

[0254] In some embodiments of this disclosure, such as Figure 13 As shown, the generation module 1204 is specifically used for:

[0255] Retrieve the image file of the training task corresponding to the model training task;

[0256] Run the training task image file and create a container environment corresponding to the model training task. The container environment is used to configure the environment information for running the model training task.

[0257] A task start message is sent to the first computing server, wherein the task start message includes a task identifier. In response to the task start message, the first computing server starts the training process corresponding to the task identifier to train the model to be trained.

[0258] In some embodiments of this disclosure, such as Figure 13 As shown, it also includes:

[0259] The sixth determining module 1211 is used to determine the priority of the model training task after controlling the first computing server to train the model to be trained based on the model training task, and when determining that resource allocation is to be carried out.

[0260] The first control module 1212 is used to control the first computing server to stop model training based on the model training task when the priority is less than or equal to the priority threshold.

[0261] In some embodiments of this disclosure, such as Figure 13 As shown, the first control module 1212 is specifically used for:

[0262] Close the container environment corresponding to the model training task; or

[0263] A process stop message is sent to the first computing server, whereby the first computing server, in response to the process stop message, stops the training process corresponding to the task identifier.

[0264] In some embodiments of this disclosure, such as Figure 13 As shown, the second determining submodule 12032 is specifically used for:

[0265] When the available resource information is greater than the first occupied resource information, a computing server with the first number of computing nodes is selected from multiple computing servers as the first computing server.

[0266] In some embodiments of this disclosure, such as Figure 13 As shown, it also includes:

[0267] The selection module 1213 is used to select a fourth number of computing nodes from the remaining computing servers as the first computing server after generating the model training task corresponding to the first computing server. The sum of the number of the first computing nodes and the number of the fourth computing nodes is less than or equal to the number of the second computing nodes. The number of the fourth computing nodes is determined by the current resource information of the server cluster.

[0268] With the above Figures 1 to 6 Corresponding to the model training method provided in the embodiments, this disclosure also provides a model training apparatus. Since the model training apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 6The model training method provided in the embodiments corresponds to the model training device provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0269] In this embodiment, by acquiring configuration information and determining the available resource information of the server cluster, and determining the first computing server from multiple computing servers based on the configuration information and available resource information, and generating a model training task corresponding to the first computing server, the configuration information is flexibly determined based on the information of the model to be trained, and the first computing server in the server cluster is flexibly selected and allocated based on the configuration information and the available resource information of the server cluster to execute the model training task based on the first computing server. This can effectively support the elastic scaling of nodes in the server cluster during the model training process, realize the flexible allocation of server cluster resources, effectively avoid resource waste, and improve model training efficiency.

[0270] Figure 14 This is a schematic diagram of the structure of a model training device proposed in one embodiment of this disclosure.

[0271] like Figure 14 As shown, the model training device 140 is applied to a server cluster, which includes multiple computing servers, including:

[0272] The first providing module 1401 is used to provide the resource manager with information about the model to be trained, wherein the information about the model to be trained is used to determine the configuration information;

[0273] The second providing module 1402 is used to provide the resource manager with information on the available resources of the server cluster;

[0274] The seventh determination module 1403 is used to determine the first computing server based on the model training task generated by the resource manager, wherein the first computing server is determined by the resource manager from multiple computing servers based on configuration information and available resource information;

[0275] The second control module 1404 is used to control the first computing server to train the model to be trained based on the model training task.

[0276] In some embodiments of this disclosure, such as Figure 15 As shown, Figure 15 This is a schematic diagram of the model training device proposed in another embodiment of this disclosure. The second control module 1404 is specifically used for:

[0277] Receive a task start message sent by the resource manager, wherein the task start message includes: task identifier;

[0278] Initiate the training process corresponding to the task identifier on the first computing server to train the model to be trained.

[0279] In some embodiments of this disclosure, such as Figure 15 As shown, the server cluster also includes: a parameter server and a data distribution server;

[0280] The second control module 1404 is specifically used for:

[0281] Control the first computing server to obtain model parameters from the parameter server;

[0282] The first computing server is controlled to obtain training sample data based on the data distribution server;

[0283] The first computing server is controlled to train the model to be trained based on the model parameters and training sample data.

[0284] In some embodiments of this disclosure, such as Figure 15 As shown, the data distribution server includes: data distribution service and data snapshot service;

[0285] The second control module 1404 is specifically used for:

[0286] The data reading location information is determined based on the data snapshot service, and then provided to the data distribution service.

[0287] The data distribution service obtains sample data corresponding to the data reading location information as training sample data.

[0288] In some embodiments of this disclosure, such as Figure 15 As shown, it also includes:

[0289] The receiving module 1405 is used to receive process stop messages sent by the resource manager;

[0290] The start / stop module 1406 is used to stop the training process corresponding to the task identifier in response to a process stop message.

[0291] With the above Figures 7 to 11 Corresponding to the model training method provided in the embodiments, this disclosure also provides a model training apparatus. Since the model training apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 7 to 11 The model training method provided in the embodiments corresponds to the model training device provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0292] In this embodiment, by providing the resource manager with information on the model to be trained and the available resources of the server cluster, and based on the model training task generated by the resource manager, a first computing server is determined and controlled. The model to be trained is then trained based on the model training task. Since the resource manager is provided with information on the model to be trained and the available resources of the server cluster, the resource manager can flexibly select and allocate the first computing server in the server cluster based on this configuration information and the available resources of the server cluster. When the model training task is executed based on the first computing server, the elastic scaling of nodes in the server cluster can be effectively supported during the model training process, enabling flexible allocation of server cluster resources, effectively avoiding resource waste, and improving model training efficiency.

[0293] To implement the above embodiments, this disclosure also proposes an electronic device, such as... Figure 16 As shown, Figure 16 This is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present disclosure. The electronic device 160 includes: the model training device 120 of the above embodiment.

[0294] To implement the above embodiments, this disclosure also proposes a server, such as... Figure 17 As shown, Figure 17 This is a schematic diagram of the structure of a server proposed in an embodiment of the present disclosure. The server 170 includes: the model training device 140 of the above embodiment.

[0295] To achieve the above embodiments, this disclosure also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it is able to achieve the functions described in this disclosure. Figures 1 to 6 The model training method proposed in the embodiments, or the implementation as described above in this disclosure. Figures 7 to 11 The proposed model training method in the example.

[0296] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the functions described in the foregoing disclosure. Figures 1 to 6 The model training method proposed in the embodiments, or the implementation as described above in this disclosure. Figures 7 to 11 The proposed model training method in the example.

[0297] To implement the above embodiments, this disclosure also proposes a computer program product that, when executed by an instruction processor, performs the functions described in this disclosure. Figures 1 to 6 The model training method proposed in the embodiments, or the method described above in this disclosure, can be performed. Figures 7 to 11 The proposed model training method in the example.

[0298] Figure 18 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 18 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0299] like Figure 18 As shown, electronic device 12 is represented in the form of a general-purpose computing device. Components of electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 19 connecting different system components (including system memory 28 and processing unit 16). Bus 19 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the MicroChannel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0300] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0301] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 18 Not shown; usually referred to as a "hard drive".

[0302] although Figure 18Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 19 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0303] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0304] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 19. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0305] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the model training method mentioned in the foregoing embodiments.

[0306] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0307] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0308] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0309] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0310] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0311] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0312] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0313] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0314] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0315] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A model training method, characterized in that, Applied to a resource manager that manages the resources of a server cluster, the method includes: Obtain configuration information, wherein the configuration information is determined by the information of the model to be trained, and the configuration information includes: the number of computing nodes; Determine the available resource information of the server cluster; The resource information occupied by the computing servers with the number of computing nodes is determined, wherein the resource information includes: first resource information corresponding to the computing servers with a first number of computing nodes, and second resource information corresponding to the computing servers with a second number of computing nodes, wherein the first number of computing nodes is less than the second number of computing nodes; If the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the first computing server is determined from multiple computing servers based on the available resource information; If the available resource information is greater than the second occupied resource information, a first computing server is determined from a plurality of computing servers based on the second occupied resource information, wherein the plurality of computing servers belong to the server cluster; and Generate a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained; The method further includes: If the available resource information is less than or equal to the first occupied resource information, then the current resource information of the server cluster is determined; The available resource information is updated based on the current resource information.

2. The method as described in claim 1, characterized in that, The step of determining the first computing server from the plurality of computing servers based on the available resource information includes: Determine the number of third computing nodes supported by the available resource information; From the plurality of computing servers, a computing server with the third number of computing nodes is selected as the first computing server.

3. The method as described in claim 1, characterized in that, The step of determining the first computing server from the plurality of computing servers based on the second resource occupancy information includes: From the plurality of computing servers, a computing server with the second number of computing nodes is selected as the first computing server.

4. The method as described in claim 1, characterized in that, The configuration information also includes: the number of parameter nodes and domain name configuration information; wherein, before generating the model training task corresponding to the first computing server, it also includes: The parameter identifier that determines the number of parameter nodes; Determine the first computing identifier of the first computing server; Configure the parameter server to which the parameter identifier belongs based on the domain name configuration information, wherein the parameter server belongs to the server cluster; Configure the first computing server according to the domain name configuration information.

5. The method as described in claim 4, characterized in that, The generation of the model training task corresponding to the first computing server includes: The first computing server is controlled to train the model to be trained based on the model training task, wherein the first computing server obtains model parameters from the parameter server to train the model to be trained.

6. The method as described in claim 5, characterized in that, The control of the first computing server to train the model to be trained based on the model training task includes: Retrieve the training task image file corresponding to the model training task; Run the training task image file to create a container environment corresponding to the model training task, wherein the container environment is used to configure the environment information for running the model training task; A task start message is sent to the first computing server, wherein the task start message includes a task identifier. In response to the task start message, the first computing server starts a training process corresponding to the task identifier to train the model to be trained.

7. The method as described in claim 6, characterized in that, After controlling the first computing server to train the model to be trained based on the model training task, the method further includes: If it is determined that resource allocation is required, then the priority of the model training task is determined; If the priority is less than or equal to the priority threshold, then the first computing server is controlled to stop model training based on the model training task.

8. The method as described in claim 7, characterized in that, The step of controlling the first computing server to stop model training based on the model training task includes: Close the container environment corresponding to the model training task; or A process stop message is sent to the first computing server, wherein the first computing server, in response to the process stop message, stops the training process corresponding to the task identifier.

9. A model training method, characterized in that, Applied to a server cluster, the server cluster comprising: multiple computing servers, wherein the method includes: Provide the resource manager with information on the model to be trained, wherein the information on the model to be trained is used to determine configuration information, the configuration information including: the number of computing nodes; Provide the resource management party with information on the available resources of the server cluster; Based on the model training task generated by the resource manager, a first computing server is determined. This first computing server is determined by the resource manager from among the plurality of computing servers based on the configuration information and the available resource information. The resource manager determines the resource information occupied by the computing servers with the specified number of computing nodes. This occupied resource information includes: first occupied resource information corresponding to the first number of computing nodes and second occupied resource information corresponding to the second number of computing nodes, where the first number of computing nodes is less than the second number of computing nodes. Furthermore, if the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the resource manager determines the first computing server from among the plurality of computing servers based on the available resource information. If the available resource information is greater than the second occupied resource information, the resource manager determines the first computing server from among the plurality of computing servers based on the second occupied resource information. The plurality of computing servers belong to the server cluster. Finally, if the available resource information is less than or equal to the first occupied resource information, the resource manager determines the current resource information of the server cluster and updates the available resource information based on the current resource information. Control the first computing server to train the model to be trained based on the model training task.

10. The method as described in claim 9, characterized in that, The control of the first computing server to train the model to be trained based on the model training task includes: Receive a task start message sent by the resource manager, wherein the task start message includes: a task identifier; In the first computing server, a training process corresponding to the task identifier is started to train the model to be trained.

11. The method as described in claim 9, characterized in that, The server cluster also includes: a parameter server and a data distribution server; Wherein, controlling the first computing server to train the model to be trained based on the model training task includes: The first computing server is controlled to obtain model parameters from the parameter server; The first computing server is controlled to acquire training sample data based on the data distribution server; The first computing server is controlled to train the model to be trained based on the model parameters and the training sample data.

12. The method as described in claim 11, characterized in that, The data distribution server includes: a data distribution service and a data snapshot service; Wherein, obtaining training sample data based on the data distribution server includes: The data reading location information is determined based on the data snapshot service, and the data reading location information is provided to the data distribution service; The data distribution service obtains sample data corresponding to the data reading location information as the training sample data.

13. The method as described in claim 10, characterized in that, Also includes: Receive the process stop message sent by the resource manager; In response to the process stop message, the training process corresponding to the task identifier is stopped.

14. A model training device, characterized in that, An apparatus for use in a resource management system that manages the resources of a server cluster, wherein the apparatus includes: An acquisition module is used to acquire configuration information, wherein the configuration information is determined by the information of the model to be trained, and the configuration information includes: the number of computing nodes; The first determining module is used to determine the available resource information of the server cluster; The second determining module is used to determine the resource information occupied by the computing servers of the specified number of computing nodes. The resource information includes: first resource information corresponding to a first number of computing servers and second resource information corresponding to a second number of computing servers, wherein the first number of computing nodes is less than the second number of computing nodes. If the available resource information is greater than the first resource information and less than or equal to the second resource information, a first computing server is determined from a plurality of computing servers based on the available resource information. If the available resource information is greater than the second resource information, a first computing server is determined from a plurality of computing servers based on the second resource information, wherein the plurality of computing servers belong to the server cluster. A generation module is used to generate a model training task corresponding to the first computing server, wherein the model training task is used to trigger the first computing server to train the model to be trained. The third determining module is used to: determine the current resource information of the server cluster if the available resource information is less than or equal to the first occupied resource information; The update module is used to update the available resource information based on the current resource information.

15. A model training device, characterized in that, Applied to a server cluster, the server cluster comprising: multiple computing servers, wherein the device includes: The first providing module is used to provide the resource manager with information about the model to be trained, wherein the information about the model to be trained is used to determine configuration information, and the configuration information includes: the number of computing nodes; The second providing module is used to provide the resource management party with information on the available resources of the server cluster; The seventh determining module is used to determine a first computing server based on the model training task generated by the resource manager. The first computing server is determined by the resource manager from among the plurality of computing servers based on the configuration information and the available resource information. The resource manager determines the resource-occupied information of the computing servers with the specified number of computing nodes. The occupied resource information includes: first occupied resource information corresponding to the first number of computing nodes and second occupied resource information corresponding to the second number of computing nodes, where the first number of computing nodes is less than the second number of computing nodes. Furthermore, if the available resource information is greater than the first occupied resource information and less than or equal to the second occupied resource information, the resource manager determines the first computing server from among the plurality of computing servers based on the available resource information. If the available resource information is greater than the second occupied resource information, the resource manager determines the first computing server from among the plurality of computing servers based on the second occupied resource information. The plurality of computing servers belong to the server cluster. Finally, if the available resource information is less than or equal to the first occupied resource information, the resource manager determines the current resource information of the server cluster and updates the available resource information based on the current resource information. The second control module is used to control the first computing server to train the model to be trained based on the model training task.

16. An electronic device comprising: The model training apparatus as described in claim 14.

17. A server, comprising: The model training apparatus as described in claim 15.

18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the model training method of any one of claims 1-8, or to perform the model training method of any one of claims 9-13.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the model training method of any one of claims 1-8, or to execute the model training method of any one of claims 9-13.

Images