Server cluster processing method and system and electronic equipment

Abstract

The invention discloses a server cluster processing method and system and electronic equipment. The method relates to the field of data processing in the field of artificial intelligence, and comprises the following steps: obtaining traffic data of a plurality of nodes in a server cluster, the server cluster being used for executing training tasks of a plurality of neural network models, the flow data is used for representing data transmitted by network communication among a plurality of nodes in the process of executing the training task by the server cluster; based on the traffic data of any node, a training period corresponding to any node is determined, and the training period is used for representing a period of executing a corresponding training task by any node; the multiple nodes are clustered based on the training periods corresponding to the multiple nodes, multiple node sets are obtained, and the nodes in the same node set are used for executing the training task of the same neural network model. The technical problem that the accuracy of distinguishing nodes executing different training tasks in a server cluster in the prior art is low is solved.

Description

Technical Field

[0001] This application relates to the field of data processing in the field of artificial intelligence, and more specifically, to a method, system, and electronic device for processing server clusters. Background Technology

[0002] In neural network model training scenarios, pre-deployed server clusters are typically used to perform distributed training of neural network models based on training tasks. This means that multiple neural network models may be trained simultaneously using the same server cluster. To ensure that the same server cluster can successfully and efficiently train multiple neural network models, it is necessary to monitor the training status of the server cluster.

[0003] Currently, monitoring the training status of server clusters typically involves acquiring the training data for each neural network model and analyzing this data along with the individual nodes (i.e., individual servers) within the cluster to monitor the training progress. However, because multiple training tasks may exist simultaneously within a server cluster, and these tasks are executed by multiple nodes with complex relationships between them, the accuracy in distinguishing which nodes are performing different training tasks is low. This makes monitoring the execution of training tasks across the server cluster quite challenging.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, system, and electronic device for processing server clusters, to at least solve the technical problem of low accuracy in distinguishing nodes performing different training tasks in a server cluster in related technologies.

[0006] According to one aspect of the embodiments of this application, a server cluster processing method is provided, comprising: acquiring traffic data of multiple nodes in the server cluster, wherein the server cluster is used to execute training tasks of multiple neural network models, and the traffic data is used to characterize the data transmitted between multiple nodes through network communication during the execution of training tasks in the server cluster; determining the training period corresponding to any node based on the traffic data of any node, wherein the training period is used to characterize the period during which any node executes the corresponding training task; clustering the multiple nodes based on the training periods corresponding to the multiple nodes to obtain multiple node sets, wherein nodes in the same node set are used to execute the training task of the same neural network model.

[0007] According to another aspect of the embodiments of this application, a server cluster processing method is also provided, comprising: obtaining traffic data of multiple nodes in the server cluster by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the traffic data of multiple nodes, the server cluster is used to execute training tasks of multiple neural network models, and the traffic data is used to characterize the data transmitted by multiple nodes through network communication during the execution of training tasks in the server cluster; determining the training period corresponding to any node based on the traffic data of any node, wherein the training period is used to characterize the period during which any node executes the corresponding training task; clustering multiple nodes based on the training periods corresponding to multiple nodes to obtain multiple node sets, wherein nodes in the same node set are used to execute the training task of the same neural network model; and outputting multiple node sets by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter includes the multiple node sets.

[0008] According to another aspect of the embodiments of this application, a server cluster processing system is also provided, comprising: a server cluster including multiple nodes, the server cluster being used to execute training tasks of multiple neural network models; a monitoring device connected to the multiple nodes, used to determine the training period corresponding to any node based on the traffic data of any node, and to cluster the multiple nodes based on the training periods corresponding to the multiple nodes to obtain multiple node sets, wherein the traffic data is used to characterize the data transmitted between the multiple nodes through network communication during the execution of training tasks by the server cluster, the training period is used to characterize the period during which any node executes the corresponding training task, and the nodes in the same node set are used to execute the training task of the same neural network model.

[0009] According to another aspect of the embodiments of this application, a processing apparatus for a server cluster is also provided, comprising: an acquisition module, configured to acquire traffic data of multiple nodes in the server cluster, wherein the server cluster is used to execute training tasks of multiple neural network models, and the traffic data is used to characterize the data transmitted between multiple nodes through network communication during the execution of training tasks in the server cluster; a determination module, configured to determine the training period corresponding to any node based on the traffic data of any node, wherein the training period is used to characterize the period during which any node executes the corresponding training task; and a clustering module, configured to cluster multiple nodes based on the training periods corresponding to multiple nodes to obtain multiple node sets, wherein nodes in the same node set are used to execute the training task of the same neural network model.

[0010] According to another aspect of the embodiments of this application, a processing apparatus for a server cluster is also provided, comprising: an acquisition module, configured to acquire traffic data of multiple nodes in the server cluster by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the traffic data of multiple nodes, the server cluster is used to execute training tasks of multiple neural network models, and the traffic data is used to characterize the data transmitted by the multiple nodes through network communication during the execution of the training tasks of the server cluster; a determination module, configured to determine the training period corresponding to any node based on the traffic data of any node, wherein the training period is used to characterize the period during which any node executes the corresponding training task; a clustering module, configured to cluster the multiple nodes based on the training periods corresponding to the multiple nodes to obtain multiple node sets, wherein nodes in the same node set are used to execute the training task of the same neural network model; and an output module, configured to output the multiple node sets by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter includes the multiple node sets.

[0011] According to another aspect of the embodiments of this application, a computer terminal is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0013] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0016] In this embodiment, the method involves acquiring traffic data from multiple nodes in a server cluster; determining the training period corresponding to any node based on the traffic data of any given node; and clustering multiple nodes based on their corresponding training periods to obtain multiple node sets. It is noteworthy that the node traffic data is general traffic data generated by network communication between nodes, meaning it is relatively easy to obtain. Furthermore, by analyzing this readily available traffic data, it eliminates the need to acquire training data for specific training tasks. Moreover, by clustering the nodes corresponding to the traffic data, different nodes processing the same neural network model can be grouped into the same node set. Different node sets correspond to different training tasks, thus clearly distinguishing nodes performing different training tasks. This achieves the technical effect of accurately identifying nodes performing different training tasks, reducing the difficulty of monitoring the execution of training tasks in the server cluster, and thereby solving the technical problem of low accuracy in distinguishing nodes performing different training tasks in related technologies.

[0017] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a server cluster processing method according to an embodiment of this application.

[0020] Figure 2 This is a structural block diagram of a computing environment according to an embodiment of this application;

[0021] Figure 3 This is a structural block diagram of a service mesh according to an embodiment of this application;

[0022] Figure 4 This is a flowchart of a server cluster processing method according to Embodiment 1 of this application;

[0023] Figure 5 This is a schematic diagram of an optional method of clustering multiple nodes in a server cluster to obtain multiple node sets according to Embodiment 1 of this application;

[0024] Figure 6 This is a flowchart of a server cluster processing method according to Embodiment 2 of this application;

[0025] Figure 7 This is a schematic diagram of a server cluster processing system according to Embodiment 3 of this application;

[0026] Figure 8 This is a schematic diagram of a server cluster processing device according to Embodiment 4 of this application;

[0027] Figure 9 This is a schematic diagram of a server cluster processing device according to Embodiment 5 of this application;

[0028] Figure 10 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0032] Large-scale models with multiple neural networks: These are models trained using large amounts of data and complex algorithms, and can be used for various tasks such as prediction, classification, and clustering.

[0033] Distributed training refers to the process of breaking down a training task into multiple sub-tasks, training them on different computers, and then merging the results.

[0034] Monitoring view: This refers to the view used to monitor the operating status of the system, which can display various indicators and alarm information.

[0035] Alias ​​curves: These are curves used to represent the relationship between multiple datasets and can be used to discover similarities and differences between datasets.

[0036] Clustering: refers to the process of grouping objects in a dataset, which can be used to discover the inherent structure and patterns in the dataset.

[0037] Traffic cycle: refers to the periodic changes in network traffic, which can be used to predict network traffic trends and optimize the allocation of network resources.

[0038] Fourier transform: refers to the process of converting a signal into a frequency domain representation, which can be used to analyze the frequency components of a signal and extract signal features.

[0039] Fundamental frequency: The lowest frequency periodic component in a periodic signal, which can be used to extract signal features and identify signal type.

[0040] Time complexity refers to the relationship between the execution time of an algorithm and the size of the input. It can be used to evaluate the efficiency of an algorithm and optimize its performance.

[0041] DBSCAN (Density-Based Spatial Clustering of Applications with Noise): A density-based clustering algorithm that can be used to discover dense regions and clusters in a dataset.

[0042] Example 1

[0043] According to an embodiment of this application, a method for processing a server cluster is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a server cluster processing method according to an embodiment of this application. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0045] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the method in the above embodiments. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0048] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0049] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computer terminal 10 (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The computer terminal 10 (or mobile device) shown is an embodiment of a computing node in computing environment 201. Figure 2 This is a structural block diagram of a computing environment according to an embodiment of this application, such as... Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (represented as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0050] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0051] The services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0052] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, such as Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers within a Pod handle requests related to one or more corresponding functions of the service. Proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.

[0053] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0054] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0055] In another alternative embodiment, Figure 3 The use of the above is illustrated in a block diagram. Figure 1 The computer terminal 10 (or mobile device) shown is an embodiment of a service mesh. Figure 3 This is a structural block diagram of a service mesh according to an embodiment of this application, such as... Figure 3 As shown, the service mesh 300 is mainly used to facilitate secure and reliable communication between multiple microservices. Microservices refer to the decomposition of an application into multiple smaller services or instances, which are distributed across different clusters / machines.

[0056] like Figure 3 As shown, a microservice may include application service instance A and application service instance B, which together form the functional application layer of service mesh 300. In one implementation, application service instance A runs as a container / process 308 on machine / workload container group 314 (Pod), and application service instance B runs as a container / process 310 on machine / workload container group 316 (Pod).

[0057] In one implementation, application service instance A can be a product query service, and application service instance B can be a product order placement service.

[0058] like Figure 3 As shown, application service instance A and grid agent (sidecar) 303 coexist in machine workload container group 314, and application service instance B and grid agent 305 coexist in machine workload container 316. Grid agents 303 and 305 form the data plane layer of service mesh 300. Grid agents 303 and 305 run as containers / processes 304 and 306 respectively, and can receive requests 312 for product query services. Grid agent 303 and application service instance A can communicate bidirectionally, and grid agent 305 and application service instance B can also communicate bidirectionally. Furthermore, grid agents 303 and 305 can also communicate bidirectionally with each other.

[0059] In one implementation, traffic from application service instance A is routed to the appropriate destination via mesh proxy 303, and network traffic from application service instance B is routed to the appropriate destination via mesh proxy 305. It should be noted that the network traffic mentioned here includes, but is not limited to, Hypertext Transfer Protocol (HTTP), Representational State Transfer (REST), high-performance, general-purpose open-source frameworks (Google Remote Procedure Call, gRPC), and open-source in-memory data structure storage systems (Redis).

[0060] In one implementation, the functionality of the extended data plane layer can be achieved by writing custom filters for the agents (Envoy) in service mesh 300. The service mesh agent configuration can enable the service mesh to correctly proxy service traffic, achieving service interoperability and service governance. Mesh agents 303 and 305 can be configured to perform at least one of the following functions: service discovery, health checking, routing, load balancing, authentication and authorization, and observability.

[0061] like Figure 3 As shown, the service mesh 300 also includes a control plane layer. This control plane layer can consist of a set of services running in a dedicated namespace, hosted by a managed control plane component 301 within a machine / workload container group (machine / Pod) 302. Figure 3 As shown, the managed control plane component 301 communicates bidirectionally with grid agents 303 and 305. The managed control plane component 301 is configured to perform several control and management functions. For example, the managed control plane component 301 receives telemetry data transmitted by grid agents 303 and 305 and can further aggregate this telemetry data. In addition to these services, the managed control plane component 301 can also provide a user-facing application programming interface (API) to facilitate easier manipulation of network behavior and to provide configuration data to grid agents 303 and 305.

[0062] Under the aforementioned operating environment, this application provides the following: Figure 4 The server cluster processing method is shown. Figure 4 This is a flowchart of a server cluster processing method according to Embodiment 1 of this application, as shown below. Figure 4 As shown, the system includes a server cluster 20 and a specific server 10, which are connected via a network. Server cluster 10 performs the following actions: reporting traffic data from multiple nodes; specific server 10 performs the following actions: acquiring traffic data from multiple nodes in the server cluster; determining the training period corresponding to any given node based on the traffic data of any single node; and clustering the multiple nodes based on their corresponding training periods to obtain a set of multiple nodes. Figure 4 As shown, the method includes the following steps:

[0063] Step S402: Obtain traffic data of multiple nodes in the server cluster. The server cluster is used to execute training tasks for multiple neural network models, and the traffic data is used to characterize the data transmitted between multiple nodes during network communication during the execution of training tasks in the server cluster.

[0064] The aforementioned server cluster can train multiple neural network models based on training tasks. The server cluster can consist of multiple local servers, multiple edge servers, or multiple cloud servers; the specific configuration of the server cluster is not limited in this embodiment. The multiple neural network models can be neural network models with relatively large parameters that require distributed training. These multiple neural network models are used for various tasks such as prediction, classification, and clustering. The application scenarios for these multiple neural network models can include, but are not limited to, natural language processing, computer vision processing, reinforcement learning, and speech recognition and synthesis. The types of tasks performed by the multiple neural network models differ depending on the application scenario.

[0065] The aforementioned training tasks can be pre-defined by the user. Based on these tasks, multiple neural network models can be trained into the user's desired neural network models capable of handling specific tasks. These training tasks may include, but are not limited to: data management, memory and computational resource optimization, mixed-precision training, model fine-tuning, hyperparameter optimization, model pruning and quantization, model distillation, model robustness testing, model evaluation, and environment and dependency management. During the training of multiple neural network models based on these training tasks, network communication between multiple nodes in the server cluster generates traffic data corresponding to the training tasks. The traffic data varies depending on the specific training task. In this embodiment, the traffic data may include, but is not limited to: model parameter synchronization, gradient information, weights and biases, optimizer status, loss function values, performance metrics, control commands, logs and monitoring data, data sharding, model checkpoints, resource scheduling information, configuration and parameter updates, and error and exception information.

[0066] In one optional embodiment, current methods for training neural network models face challenges in monitoring the training status of server clusters and accurately distinguishing between different training tasks. To address these issues, the method proposed in this application, when monitoring the training status of a server cluster, firstly acquires general traffic data and the volume of traffic data from multiple nodes within the server cluster. The server cluster is used to execute training tasks for multiple neural network models, and the traffic data characterizes the data transmitted via network communication during the execution of these training tasks. For example, when monitoring the training status of multiple neural network models in a natural language processing scenario, based on memory and computing resource optimization (i.e., the training task), a server cluster composed of multiple cloud servers can first acquire the traffic data generated by network communication between multiple nodes processing this training task within the server cluster, as well as the volume of traffic data. However, this is not the only possibility. For example, when monitoring a server cluster consisting of multiple edge servers in a computer vision processing scenario, and observing the training of multiple neural network models based on model robustness testing (i.e., the training task), one can first obtain the network traffic data generated by the network communication between the multiple nodes processing the training task in the server cluster, as well as the amount of traffic data, but this is not the only option. It should be noted that different training tasks will correspond to different traffic data and different amounts of traffic data.

[0067] Step S404: Based on the traffic data of any node, determine the training period corresponding to any node, wherein the training period is used to characterize the period during which any node executes the corresponding training task.

[0068] In one optional embodiment, given the traffic data of any node, the training period corresponding to that node can be determined based on the traffic data of that node. For example, the traffic data can first be preprocessed, such as by cleaning and normalization, to obtain preprocessed traffic data. Then, features related to the training period, such as traffic peaks and fluctuations, can be extracted from the preprocessed traffic data. These extracted features can then be input into the trained periodic processing model to obtain the training period corresponding to any node. Alternatively, a Fourier transform can be performed on the traffic data corresponding to any node to obtain its spectrum, and the training period can be obtained based on this spectrum, but this is not limited to these methods.

[0069] Step S406: Cluster multiple nodes based on the training cycles corresponding to multiple nodes to obtain multiple node sets, wherein nodes in the same node set are used to perform the training task of the same neural network model.

[0070] In one optional embodiment, after obtaining multiple training periods corresponding to multiple nodes, the multiple nodes can be clustered based on the training periods to obtain multiple node sets. For example, after obtaining multiple training periods, multiple nodes with the same training period can be clustered to obtain multiple node sets, wherein multiple nodes in the same node set have the same training period, and multiple nodes in different node sets have different training periods. As another example, after obtaining multiple training periods corresponding to multiple nodes, the training periods can be classified based on different period length ranges, and then the nodes corresponding to the classified training periods can be clustered to obtain multiple node sets, wherein the training periods corresponding to multiple nodes in the same node set are within the same period length range. For example, when obtaining training periods corresponding to multiple nodes, the multiple training periods can be classified based on period length ranges such as [0,1], (1,3], (3,5], etc., and then the nodes corresponding to the classified training periods can be clustered to obtain multiple node sets.

[0071] In this embodiment, the method involves acquiring traffic data from multiple nodes in a server cluster; determining the training period corresponding to any node based on the traffic data of any given node; and clustering multiple nodes based on their corresponding training periods to obtain multiple node sets. It is noteworthy that the node traffic data is general traffic data generated by network communication between nodes, meaning it is relatively easy to obtain. Furthermore, by analyzing this readily available traffic data, it eliminates the need to acquire training data for specific training tasks. Moreover, by clustering the nodes corresponding to the traffic data, different nodes processing the same neural network model can be grouped into the same node set. Different node sets correspond to different training tasks, thus clearly distinguishing nodes performing different training tasks. This achieves the technical effect of accurately identifying nodes performing different training tasks, reducing the difficulty of monitoring the execution of training tasks in the server cluster, and thereby solving the technical problem of low accuracy in distinguishing nodes performing different training tasks in related technologies.

[0072] In the above embodiments of this application, determining the training period corresponding to any node based on the traffic data of any node includes: performing a Fourier transform on the traffic data of any node to obtain a spectrum; extracting the fundamental frequency from the spectrum; and determining the training period based on the fundamental frequency.

[0073] The fundamental frequency mentioned above can be the lowest frequency periodic component in a periodic signal, which can be used to extract signal features and identify signal types.

[0074] In one optional embodiment, after obtaining the traffic data corresponding to any node, a Fourier transform can be performed on the traffic data corresponding to any node to obtain the spectrum corresponding to any node. Then, the maximum peak value can be determined from the spectrum, and the frequency corresponding to the maximum peak value can be determined as the fundamental frequency. Finally, the training period can be obtained based on the fundamental frequency.

[0075] In the above embodiments of this application, extracting the fundamental frequency from the spectrum and determining the training period based on the fundamental frequency includes: extracting the fundamental frequency from the spectrum; performing a reciprocal transformation on the fundamental frequency to obtain an initial period; and performing a rounding operation on the initial period to obtain the training period.

[0076] In one optional embodiment, after obtaining the spectrum corresponding to any node, the fundamental frequency can first be extracted from the spectrum. Then, the fundamental frequency can be transformed by its reciprocal to obtain the initial period. Finally, the initial period can be rounded to obtain the training period. For example, if the initial period is 1.5 times the reciprocal of the fundamental frequency, 1.5 can be rounded to obtain a training period of 2 times the reciprocal of the fundamental frequency, but this is not the only possibility. It should be noted that the values ​​in this embodiment are merely examples, and specific values ​​are not limited in this embodiment.

[0077] In the above embodiments of this application, multiple nodes are clustered based on the training periods corresponding to multiple nodes to obtain multiple node sets, including: clustering nodes corresponding to the same training period into the same node set, and clustering nodes corresponding to different training periods into different node sets.

[0078] In one optional embodiment, after obtaining multiple training cycles corresponding to multiple nodes, nodes corresponding to multiple training cycles with the same cycle length can be aggregated according to the cycle length to obtain multiple node sets. Among them, multiple nodes in the same node set have the same training cycle, and multiple nodes in different node sets have different training cycles.

[0079] In the above embodiments of this application, obtaining traffic data of multiple nodes in a server cluster includes: deploying traffic collection devices on multiple nodes to collect traffic data of multiple nodes; or, determining the target port where multiple nodes are connected to a data forwarding device, and collecting the data transmitted through the target port to obtain traffic data of multiple nodes.

[0080] The aforementioned traffic acquisition device can be a network interface card (NIC) deployed on a node of a server cluster. A program also deployed on a node of the server cluster can collect traffic information from the NIC, thus obtaining traffic data. The aforementioned data forwarding device can be a switch.

[0081] In one alternative embodiment, traffic information of multiple network interface cards (NICs) deployed on multiple nodes can be collected by multiple programs deployed on multiple nodes, thus obtaining traffic data for multiple nodes.

[0082] In another alternative embodiment, multiple target ports connected to the switch can be identified, and then the data transmitted by the target ports can be collected by a program deployed on the nodes, thus obtaining the traffic data corresponding to multiple nodes.

[0083] In the above embodiments of this application, the method further includes: determining the traffic data of at least one node in the same node set; comparing the traffic data of at least one node to determine the execution status of the training task corresponding to the same node set.

[0084] The execution status mentioned above may include, but is not limited to: execution completed, execution in progress, and execution exception.

[0085] In one optional embodiment, when traffic data of at least one node is obtained, the traffic data of the at least one node can be compared with the traffic data of a normally completed training task and the amount of traffic data. If the amount of traffic data of the at least one node is equal to the amount of traffic data of a normally completed task, and the content of the traffic data of the at least one node is the same as the content of the traffic data of a normally completed task, the execution status of the training task corresponding to the same set of nodes can be determined as completed. If the amount of traffic data of the at least one node is less than the amount of traffic data of a normally completed task, or the content of the traffic data of the at least one node is different from the content of the traffic data of a normally completed task, the execution status of the training task corresponding to the same set of nodes can be determined as in progress or an execution error has occurred. If the amount of traffic data of the at least one node is greater than the amount of traffic data of a normally completed task, or the content of the traffic data of the at least one node is different from the content of the traffic data of a normally completed task, the execution status of the training task corresponding to the same set of nodes can be determined as an execution error has occurred, but this is not limited to these cases.

[0086] In the above embodiments of this application, the method further includes: determining the communication performance between any node in the same node set and other nodes; and adjusting any node when the communication performance between any node and other nodes is less than a preset performance, so that the communication performance between any two nodes in the adjusted node set is greater than or equal to the preset performance, wherein the preset performance is related to the training task corresponding to the same node set.

[0087] The communication performance described above represents the efficiency and effectiveness of data exchange and collaborative work between nodes. The preset performance can be set by the user in advance to determine whether to adjust any particular node. The preset performance is correlated with the training task corresponding to the same set of nodes.

[0088] In an optional embodiment, the communication performance between any node in the same node set and other nodes can also be determined. If the communication performance between any node and other nodes is less than the preset performance, it indicates that the distance between the node and other nodes is too far, which will affect the communication performance. At this time, the position of any node can be adjusted so that the communication performance between any two nodes in the adjusted node set is greater than or equal to the preset performance.

[0089] It should be noted that in the adjusted node set, the distance between any two nodes is closer. For example, after adjusting the position of any node, any two nodes will be in the same rack, thus reducing the number of hops with the switch and improving the communication performance between any two nodes.

[0090] In the above embodiments of this application, the method further includes: obtaining historical traffic data of at least one node in the same node set; comparing the historical traffic data with the traffic data of at least one node to obtain a detection result of at least one node, wherein the detection result is used to characterize whether at least one node has failed.

[0091] The aforementioned historical traffic data can be traffic data generated by network communication between multiple nodes during the training of multiple neural network models by the server cluster based on historical training tasks within a historical time period.

[0092] In another optional embodiment, historical traffic data of at least one node in the same node set can be obtained. The historical traffic data can then be compared with the traffic data of at least one node to obtain a detection result for at least one node. This detection result is used to characterize whether at least one node has failed. For example, if the historical traffic data is less than or greater than the traffic data of at least one node, the detection result can be determined that at least one node has failed; if the historical traffic data is equal to the traffic data of at least one node, the detection result can be determined that at least one node has not failed, but this is not limited to these possibilities.

[0093] Figure 5 This is a schematic diagram illustrating an optional clustering method for multiple nodes in a server cluster, according to Embodiment 1 of this application, to obtain multiple node sets. This method primarily utilizes the similarity of traffic cycles among nodes during distributed training. By aggregating nodes with similar cycles together, it achieves the purpose of distinguishing different training tasks. Figure 5 As shown, you can first query and obtain the time-traffic curves of all nodes in the cluster (e.g.) Figure 5 The time series curves (1-N) can be used as examples. Secondly, Fourier transforms can be performed on multiple time series curves separately to obtain multiple frequency domain curves (such as...). Figure 5 The spectrum (1-N) in the frequency domain can be used to extract the fundamental frequency from the frequency domain curve. This frequency is the frequency of change in the flow data. Taking the reciprocal of the frequency gives the training period (e.g., ...). Figure 5 The period (1-N) can be used to aggregate multiple nodes according to the rounded period, thus obtaining clusters with different traffic characteristics (e.g., ...). Figure 5 Clustering 1-K in the data.

[0094] The effectiveness of this method is reflected in the high similarity and synchronization of the clustered curves, effectively solving the problem of aliasing in multi-task monitoring curves within a cluster. The advantages of this method are its fewer limitations; it only requires traffic metrics to complete clustering. Traffic metrics are universal, widely collected, and low-sensitivity indicators that can be collected from both the network interface card (NIC) and switch sides, thus giving this solution strong adaptability. Furthermore, this method also has the advantage of lower time complexity.

[0095] This method is based on a flow cycle-based curve clustering approach, which effectively solves the curve aliasing problem when multiple tasks run concurrently within the same cluster. This method uses common flow metrics for classification, avoiding sensitive issues and exhibiting greater adaptability. Through periodic clustering, this method reduces the algorithm's time complexity, outperforming other conventional solutions.

[0096] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0097] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0099] Example 2

[0100] According to an embodiment of this application, a method for processing a server cluster is also provided. Figure 6 This is a flowchart of a server cluster processing method according to Embodiment 2 of this application, as shown below. Figure 6 As shown, the method includes the following steps:

[0101] Step S602: Obtain traffic data of multiple nodes in the server cluster by calling the first interface. The first interface includes a first parameter, and the parameter value of the first parameter includes traffic data of multiple nodes. The server cluster is used to execute training tasks of multiple neural network models, and the traffic data is used to characterize the data transmitted between multiple nodes in network communication during the execution of training tasks in the server cluster.

[0102] Step S604: Based on the traffic data of any node, determine the training period corresponding to any node, wherein the training period is used to characterize the period during which any node executes the corresponding training task.

[0103] Step S606: Cluster multiple nodes based on the training cycles corresponding to multiple nodes to obtain multiple node sets, wherein nodes in the same node set are used to perform the training task of the same neural network model.

[0104] Step S608: Output multiple node sets by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes multiple node sets.

[0105] The first interface mentioned above could be an interface for other servers to obtain traffic data from the server cluster. The second interface mentioned above could be an interface for other servers to output multiple node sets to the server cluster.

[0106] In an optional embodiment, when it is necessary to monitor the training status of the server cluster and to distinguish different training tasks, the traffic data of multiple nodes in the server cluster can be obtained by calling the first interface first. Then, the training period corresponding to any node can be determined based on the traffic data of any node. Then, multiple nodes can be clustered based on the training periods corresponding to multiple nodes to obtain multiple node sets. Finally, multiple node sets can be output by calling the second interface.

[0107] The first interface includes a first parameter, the value of which includes traffic data of multiple nodes. The server cluster is used to execute training tasks for multiple neural network models. The traffic data is used to characterize the data transmitted between multiple nodes during network communication during the execution of training tasks in the server cluster. The training period is used to characterize the period during which any node executes the corresponding training task. Nodes in the same set of nodes are used to execute training tasks for the same neural network model. The second interface includes a second parameter, the value of which includes multiple sets of nodes.

[0108] Example 3

[0109] According to an embodiment of this application, a server cluster processing system is also provided. Figure 7 This is a schematic diagram of a server cluster processing system according to Embodiment 3 of this application, as shown below. Figure 7 As shown, the system includes a server cluster 72 and monitoring devices 74, wherein the server cluster 72 includes multiple nodes 72-1. Figure 7 As shown, the monitoring device 74 is connected to multiple nodes 72-1 in the server cluster 72.

[0110] The server cluster is used to execute training tasks for multiple neural network models. The monitoring device is used to determine the training cycle corresponding to any node based on the traffic data of any node, and to cluster multiple nodes based on the training cycles corresponding to multiple nodes to obtain multiple node sets. The traffic data is used to characterize the data transmitted between multiple nodes through network communication during the execution of training tasks by the server cluster, and the training cycle is used to characterize the cycle of any node executing the corresponding training task. Nodes in the same node set are used to execute the training task of the same neural network model.

[0111] Example 4

[0112] According to an embodiment of this application, a server cluster processing apparatus for implementing the above-described server cluster processing method is also provided. Figure 8 This is a schematic diagram of a server cluster processing device according to Embodiment 4 of this application, as shown below. Figure 8As shown, the device includes: an acquisition module 82, a determination module 84, and a clustering module 86.

[0113] The acquisition module is used to acquire traffic data from multiple nodes in the server cluster, where the server cluster is used to execute training tasks for multiple neural network models, and the traffic data is used to characterize the data transmitted between multiple nodes during network communication during the execution of training tasks in the server cluster; the determination module is used to determine the training period corresponding to any node based on the traffic data of any node, where the training period is used to characterize the period during which any node executes the corresponding training task; the clustering module is used to cluster multiple nodes based on the training periods corresponding to multiple nodes to obtain multiple node sets, where nodes in the same node set are used to execute the training task of the same neural network model.

[0114] It should be noted that the acquisition module 82, determination module 84, and clustering module 86 mentioned above correspond to steps S402 to S406 in Embodiment 1. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0115] In the above embodiments of this application, the determining module includes a transformation unit and a determining unit.

[0116] The transformation unit is used to perform Fourier transform on the traffic data of any node to obtain the spectrum; the determination unit is used to extract the fundamental frequency from the spectrum and determine the training period based on the fundamental frequency.

[0117] In the above embodiments of this application, the determining unit includes: an extraction subunit, a transformation subunit, and an integer subunit.

[0118] The extraction subunit is used to extract the fundamental frequency from the spectrum; the transformation subunit is used to perform a reciprocal transformation on the fundamental frequency to obtain the initial period; and the rounding subunit is used to perform a rounding operation on the initial period to obtain the training period.

[0119] In the above embodiments of this application, the clustering module includes: clustering units.

[0120] The clustering unit is used to cluster nodes corresponding to the same training period into the same node set, and to cluster nodes corresponding to different training periods into different node sets.

[0121] In the above embodiments of this application, the acquisition module includes: a first acquisition unit or a second acquisition unit.

[0122] The first acquisition unit is used to collect traffic data from multiple nodes by deploying traffic acquisition devices on multiple nodes, thereby obtaining traffic data from multiple nodes; the second acquisition unit is used to determine the target port connected to the data forwarding device of multiple nodes, and collect the data transmitted by the target port, thereby obtaining traffic data from multiple nodes.

[0123] In the above embodiments of this application, the device further includes: a data determination module and a first comparison module.

[0124] The data determination module is used to determine the traffic data of at least one node in the same set of nodes; the first comparison module is used to compare the traffic data of at least one node to determine the execution status of the training task corresponding to the same set of nodes.

[0125] In the above embodiments of this application, the device further includes a performance determination module and an adjustment module.

[0126] The performance determination module is used to determine the communication performance between any node in the same node set and other nodes; the adjustment module is used to adjust any node when the communication performance between any node and other nodes is less than the preset performance, so that the communication performance between any two nodes in the adjusted node set is greater than or equal to the preset performance, wherein the preset performance is related to the training task corresponding to the same node set.

[0127] In the above embodiments of this application, the device further includes: a data acquisition module and a second comparison module.

[0128] The data acquisition module is used to acquire historical traffic data of at least one node in the same set of nodes; the second comparison module is used to compare the historical traffic data with the traffic data of at least one node to obtain the detection result of at least one node, wherein the detection result is used to characterize whether at least one node has failed.

[0129] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0130] Example 5

[0131] According to an embodiment of this application, a server cluster processing apparatus for implementing the above-described server cluster processing method is also provided. Figure 9 This is a schematic diagram of a server cluster processing device according to Embodiment 5 of this application, as shown below. Figure 9As shown, the device includes: an acquisition module 92, a determination module 94, a clustering module 96, and an output module 98.

[0132] The acquisition module is used to acquire traffic data of multiple nodes in the server cluster by calling a first interface. The first interface includes a first parameter, the value of which includes the traffic data of multiple nodes. The server cluster is used to execute training tasks for multiple neural network models, and the traffic data represents the data transmitted between multiple nodes during network communication during the execution of training tasks. The determination module is used to determine the training period corresponding to any node based on the traffic data of any node. The training period represents the period during which any node executes the corresponding training task. The clustering module is used to cluster multiple nodes based on the training periods corresponding to multiple nodes to obtain multiple node sets. Nodes in the same node set are used to execute the same neural network model training task. The output module is used to output multiple node sets by calling a second interface. The second interface includes a second parameter, the value of which includes the multiple node sets.

[0133] It should be noted that the acquisition module 92, determination module 94, clustering module 96, and output module 98 mentioned above correspond to steps S602 to S608 in Embodiment 2. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0134] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0135] Example 6

[0136] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0137] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0138] In this embodiment, the computer terminal described above can execute the program code in the method.

[0139] Optionally, Figure 10 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 10 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1002, memory 1004, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to a radio frequency module, an audio module, and a display.

[0140] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0141] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquire traffic data from multiple nodes in a server cluster; determine the training period corresponding to any node based on the traffic data of any node; and cluster multiple nodes based on the training periods corresponding to multiple nodes to obtain multiple node sets.

[0142] Optionally, the processor may also execute program code that performs the following steps: performs a Fourier transform on the traffic data of any node to obtain the spectrum; extracts the fundamental frequency from the spectrum; and determines the training period based on the fundamental frequency.

[0143] Optionally, the processor may also execute program code that performs the following steps: extracting the fundamental frequency from the spectrum; performing a reciprocal transformation on the fundamental frequency to obtain the initial period; and performing a rounding operation on the initial period to obtain the training period.

[0144] Optionally, the processor may also execute program code that performs the following steps: clustering nodes corresponding to the same training period into the same node set, and clustering nodes corresponding to different training periods into different node sets.

[0145] Optionally, the processor may also execute program code that performs the following steps: by deploying traffic acquisition devices on multiple nodes, collecting traffic data from multiple nodes to obtain traffic data from multiple nodes; or, determining the target ports where multiple nodes are connected to the data forwarding device, and collecting the data transmitted from the target ports to obtain traffic data from multiple nodes.

[0146] Optionally, the processor may also execute program code that performs the following steps: determining the traffic data of at least one node in the same set of nodes; comparing the traffic data of at least one node to determine the execution status of the training task corresponding to the same set of nodes.

[0147] Optionally, the processor may also execute program code that performs the following steps: determining the communication performance between any node in the same node set and other nodes; and adjusting any node when the communication performance between any node and other nodes is less than a preset performance, so that the communication performance between any two nodes in the adjusted node set is greater than or equal to the preset performance, wherein the preset performance is related to the training task corresponding to the same node set.

[0148] Optionally, the processor may also execute program code that performs the following steps: acquiring historical traffic data of at least one node in the same set of nodes; comparing the historical traffic data with the traffic data of at least one node to obtain a detection result of at least one node, wherein the detection result is used to characterize whether at least one node has failed.

[0149] This application provides a method for acquiring traffic data from multiple nodes in a server cluster; determining the training period corresponding to any node based on the traffic data of any given node; and clustering multiple nodes based on their corresponding training periods to obtain multiple node sets. It is noteworthy that the node traffic data is general traffic data generated by network communication between nodes, meaning it is relatively easy to obtain. Furthermore, by analyzing this easily obtainable traffic data, it eliminates the need to acquire training data for specific training tasks. Moreover, by clustering the nodes corresponding to the traffic data, different nodes processing the same neural network model can be grouped into the same node set. Different node sets correspond to different training tasks, thus clearly distinguishing nodes performing different training tasks. This achieves the technical effect of accurately distinguishing nodes performing different training tasks, reducing the difficulty of monitoring the execution of training tasks in a server cluster, and thus solving the technical problem of low accuracy in distinguishing nodes performing different training tasks in related technologies.

[0150] It will be understood by those skilled in the art that the structure shown in the figure is merely illustrative, and the electronic device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. This figure does not limit the structure of the aforementioned electronic device. For example, electronic device A may include more or fewer components (such as a network interface, a display device, etc.) than shown in the figure, or may have a different configuration than shown in the figure.

[0151] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0152] Example 7

[0153] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0154] Optionally, in this embodiment, the storage medium may be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0155] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring traffic data of multiple nodes in a server cluster; determining the training period corresponding to any node based on the traffic data of any node; and clustering multiple nodes based on the training periods corresponding to multiple nodes to obtain multiple node sets.

[0156] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: performing a Fourier transform on the traffic data of any node to obtain a spectrum; extracting the fundamental frequency from the spectrum to obtain a training period.

[0157] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: extracting the fundamental frequency from the spectrum; performing a reciprocal transformation on the fundamental frequency to obtain an initial period; rounding the initial period and determining a training period based on the fundamental frequency.

[0158] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: clustering nodes corresponding to the same training period into the same set of nodes, and clustering nodes corresponding to different training periods into different sets of nodes.

[0159] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: collecting traffic data of multiple nodes by deploying traffic acquisition devices on multiple nodes to obtain traffic data of multiple nodes; or, determining the target port where multiple nodes are connected to a data forwarding device, and collecting the data transmitted at the target port to obtain traffic data of multiple nodes.

[0160] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining traffic data of at least one node in the same set of nodes; comparing the traffic data of at least one node to determine the execution status of the training task corresponding to the same set of nodes.

[0161] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining the communication performance between any node in the same set of nodes and other nodes; and adjusting any node so that the communication performance between any two nodes in the adjusted set of nodes is greater than or equal to the preset performance, wherein the preset performance is correlated with the training task corresponding to the same set of nodes.

[0162] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: acquiring historical traffic data of at least one node in the same set of nodes; comparing the historical traffic data with the traffic data of at least one node to obtain a detection result of at least one node, wherein the detection result is used to characterize whether at least one node has failed.

[0163] Example 8

[0164] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0165] Example 9

[0166] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0167] Example 10

[0168] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0169] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0175] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing a server cluster, characterized in that, include: Acquire traffic data from multiple nodes in a server cluster, wherein the server cluster is used to execute training tasks for multiple neural network models, and the traffic data is used to characterize the data transmitted between the multiple nodes during network communication during the execution of the training tasks by the server cluster. Based on the traffic data of any node, the training period corresponding to the arbitrary node is determined, wherein the training period is used to characterize the period during which the arbitrary node executes the corresponding training task. Based on the training cycles corresponding to the multiple nodes, the multiple nodes are clustered to obtain multiple node sets, wherein nodes in the same node set are used to perform the training task of the same neural network model.

2. The method according to claim 1, characterized in that, Determining the training period corresponding to any given node based on traffic data from any given node includes: Perform a Fourier transform on the traffic data of any one of the nodes to obtain the spectrum; The fundamental frequency is extracted from the spectrum, and the training period is determined based on the fundamental frequency.

3. The method according to claim 2, characterized in that, The step of extracting the fundamental frequency from the spectrum and determining the training period based on the fundamental frequency includes: Extract the fundamental frequency from the spectrum; The initial period is obtained by performing a reciprocal transformation on the fundamental frequency; The initial period is rounded down to obtain the training period.

4. The method according to claim 1, characterized in that, The process of clustering the multiple nodes based on their corresponding training periods to obtain multiple node sets includes: Cluster nodes corresponding to the same training period into the same node set, and cluster nodes corresponding to different training periods into different node sets.

5. The method according to claim 1, characterized in that, The acquisition of traffic data from multiple nodes in the server cluster includes: Traffic data for the multiple nodes is obtained by deploying traffic acquisition devices on the multiple nodes; or, The target ports where the multiple nodes are connected to the data forwarding device are determined, and the data transmitted through the target ports are collected to obtain the traffic data of the multiple nodes.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Determine the traffic data of at least one node in the same set of nodes; The traffic data of at least one node are compared to determine the execution status of the training task corresponding to the same set of nodes.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Determine the communication performance between any node in the same set of nodes and other nodes; If the communication performance between any node and the other nodes is less than a preset performance, the node is adjusted so that the communication performance between any two nodes in the adjusted node set is greater than or equal to the preset performance, wherein the preset performance is related to the training task corresponding to the same node set.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Retrieve historical traffic data for at least one node in the same set of nodes; The historical traffic data is compared with the traffic data of the at least one node to obtain the detection result of the at least one node, wherein the detection result is used to characterize whether the at least one node has failed.

9. A method for processing a server cluster, characterized in that, include: Traffic data of multiple nodes in a server cluster is obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the traffic data of the multiple nodes, the server cluster is used to execute training tasks of multiple neural network models, and the traffic data is used to characterize the data transmitted by the multiple nodes through network communication during the execution of the training tasks by the server cluster. Based on the traffic data of any node, the training period corresponding to the arbitrary node is determined, wherein the training period is used to characterize the period during which the arbitrary node executes the corresponding training task. Based on the training cycles corresponding to the multiple nodes, the multiple nodes are clustered to obtain multiple node sets, wherein nodes in the same node set are used to perform the training task of the same neural network model. The multiple node sets are output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the multiple node sets.

10. A server cluster processing system, characterized in that, include: A server cluster, comprising multiple nodes, is used to perform training tasks for multiple neural network models. A monitoring device, connected to the plurality of nodes, is used to determine the training period corresponding to any one node based on the traffic data of any one node, and to cluster the plurality of nodes based on the training periods corresponding to the plurality of nodes to obtain a plurality of node sets. The traffic data is used to characterize the data transmitted between the plurality of nodes during network communication during the execution of the training task by the server cluster, and the training period is used to characterize the period during which any one node executes the corresponding training task. Nodes in the same node set are used to execute the training task of the same neural network model.

11. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images