A troubleshooting method for large-scale distributed services

Abstract

The present application provides a troubleshooting method for large-scale distributed services, which includes parsing a target service; injecting networking information environment variables, and running a control method; reducing operations on multiple target services to operations on one target service; building a full mesh link; linking to the target service; freezing the running status of the target service to troubleshoot its faults; after the troubleshooting is completed, unfreezing and restoring the running status of the target service; and deleting the control method, which solves the problem of difficult fault location in a stateful distributed training scenario with multiple machines and multiple cards. The fault location is fast, and all subordinate containers can be seamlessly accessed in the management container without switching views, thereby improving access efficiency. At the same time, a health check of the entire full mesh communication link is built, which can directly link to the inside of the target container during remote access. When encountering difficult problems, the running status of the target container can be suspended, which is conducive to troubleshooting problems of the target service and speeds up troubleshooting.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a troubleshooting method for large-scale distributed services. Background Art

[0002] K8s (kubernetes) is an open source container cluster orchestration system that can realize functions such as automatic deployment, automatic expansion and reduction, and maintenance of container clusters.

[0003] In the context of AI intelligent computing, in order to train larger models and run more complex services, dynamic networking of multiple machines and multiple cards for joint training is usually required. In the process of dynamic networking, many architectures have been derived, such as the master-worker architecture, Rendezvous architecture, Parameter Server architecture, and scheduler-worker architecture. Each architecture is a specific implementation of a distributed service. What follows is how to quickly solve problems encountered by distributed services.

[0004] On the one hand, due to the inherent complexity of distributed services, it is difficult to locate faults when they occur. It is necessary to check the working status and operation log of each node and test whether the p2p communication is normal. This process is time-consuming and labor-intensive, and as the number of nodes increases, the difficulty of locating faults increases exponentially.

[0005] On the other hand, Kubernetes' native debugging tools such as spdy (or websocket) can only access a single container and cannot access multiple containers at the same time, which means that it is impossible to effectively integrate failures across the entire distributed service.

[0006] More importantly, this distributed service is stateful, and there are a huge number of worker services (10+ / 100+ / ). It is necessary to check the link status and operation log of each service at the same time and consider them comprehensively to find the root cause, which brings great challenges to troubleshooting. Summary of the invention

[0007] The present application shows a troubleshooting method for large-scale distributed services.

[0008] In a first aspect, the present application provides a method for troubleshooting large-scale distributed services, the method comprising:

[0009] Analyze the target service;

[0010] A control method based on a kubernetes application software custom resource definition extension mechanism, the control method adopts a master-slave architecture, the master-slave architecture includes a management node and a plurality of slave nodes, the management node is used to coordinate the working status of the slave nodes, inject networking information environment variables into each of the slave nodes, and run the control method;

[0011] Based on an inter-process messaging tool, the management node issues commands, and collects and aggregates results returned by all the slave nodes, so as to reduce operations on multiple target services to operations on one target service;

[0012] Building a full mesh link based on the kubernetes application software and the inter-process messaging tool;

[0013] Based on the nsenter mechanism, link to the target service;

[0014] Freeze the running state of the target service based on the freezing capability of the control group mechanism, so as to troubleshoot the failure of the target service;

[0015] After troubleshooting is completed, unfreeze and restore the operating status of the target service;

[0016] Delete the control method.

[0017] According to the method of the first aspect of the present application, the target service is parsed, and the parsed information includes the architecture type, the number of container copies, the target node name, the container name and the container ID.

[0018] According to the method of the first aspect of the present application, the environment variables injected into the network in each of the slave nodes include the target service container ID, the process ID, the service name and the port number of the management node.

[0019] According to the method of the first aspect of the present application, based on the master-slave architecture, a management container and the slave container are run, and the slave container and the target service run on the same node and have the same number of replicas.

[0020] According to the method of the first aspect of the present application, the management node issues a command and collects and aggregates the results returned by all the slave nodes, including:

[0021] The log information, process information, network information, CPU and memory information, RDMA network card, GPU status and NPU status of the target container in the kubernetes application software are transferred out.

[0022] According to the method of the first aspect of the present application, the construction of a full mesh link based on the kubernetes application software and the inter-process messaging tool includes:

[0023] Based on the service discovery capability of the kubernetes application software and the distributed capability of the inter-process messaging tool, during each traversal process, one of the slave nodes is selected as the server, and the other slave nodes are selected as the client;

[0024] The traversal is not completed until the last subordinate node is reached.

[0025] According to the method of the first aspect of the present application, the linking to the target service based on the nsenter mechanism includes:

[0026] Granting the slave container runtime privileges is true, and network privileges are true;

[0027] Directly link to the namespace where the target program is located;

[0028] Use the above method as the entrypoint command to directly link to the target container during remote access.

[0029] According to the method of the first aspect of the present application, the freezing capability of the control group mechanism, freezing the running state of the target service, includes:

[0030] The freezer.state of the target process of the target service is set to FROZEN to facilitate observation of the running state of the internal process of the target service.

[0031] According to the method of the first aspect of the present application, after the troubleshooting is completed, unfreezing and restoring the operating state of the target service includes:

[0032] Set the target process's freezer.state to THAWED.

[0033] According to the method of the first aspect of the present application, the network privileges include hostPID, hostIPC and hostNetwork.

[0034] The technical solution provided by this application may have the following beneficial effects:

[0035] This application solves the problem of difficult fault location in a multi-machine, multi-card stateful distributed training scenario: by optimizing the command issuance and collection steps, the number of manual accesses is reduced, and the operation of multiple targets is reduced to the operation of one target. All subordinate containers can be seamlessly accessed in the management container without switching views, which improves access efficiency. At the same time, the service discovery capabilities of the kubernetes application software and the distributed capabilities of the inter-process messaging tool are used to build a health check for the entire full-mesh communication link. Further configure the subordinate container as a privileged container, use the nsenter mechanism to directly link to the namespace where the target program is located, and use this method as the entrypoint command to facilitate remote access and directly link to the inside of the target container. Finally, when encountering difficult problems or abnormal containers, it is necessary to suspend the running status of the target container, which is conducive to troubleshooting the target service, which is also a very effective method for containers. Through these steps, the speed of troubleshooting in a multi-machine, multi-card stateful distributed training scenario can be accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of the steps of a troubleshooting method for large-scale distributed services in the present application.

[0037] Figure 2 It is an overall schematic diagram of the troubleshooting method in this application.

[0038] Figure 3 It is a block diagram of an electronic device of the present application.

[0039] Figure 4 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Before introducing the troubleshooting method for large-scale distributed services provided by this application, the professional terms involved in this application are first explained.

[0042] K8s (kubernetes): It is an open source container cluster orchestration system that can realize the automatic deployment, automatic expansion and reduction, and maintenance of container clusters.

[0043] CRD: Kubernetes Custom Resource Definition (CRD) is a powerful Kubernetes API extension mechanism. It allows users to create and manage custom resources.

[0044] Operator: The control of custom resource definition (CRD) is called Operator. Kubernetes Operator is a method of encapsulating, deploying and managing Kubernetes applications. We use the Kubernetes API (application programming interface) and kubectl tool to deploy and manage Kubernetes applications on Kubernetes.

[0045] OpenMPI is a high-performance message passing library and an inter-process message passing tool. It was originally developed as a fusion of technologies and resources from several other projects (FT-MPI, LA-MPI, LAM / MPI, and PACX-MPI). It is an open source implementation of the MPI-2 standard, developed and maintained by some scientific research institutions and enterprises.

[0046] The nsenter mechanism is a tool used in Linux systems to enter the namespace of a specific process and execute commands. It is located in the util-linux package and is mainly used in container and virtualization environments to help users debug and operate containers using host commands without entering the container.

[0047] Cgroup, whose name comes from the abbreviation of control groups, is a function of the Linux kernel that is used to limit, control and separate the resources of a process group (such as CPU, memory, disk input and output, etc.). This project was first initiated by Google engineers (mainly Paul Menage and Rohit Seth) in 2006, and its original name was process containers. In 2007, because the term container has many different meanings in the Linux kernel, it was renamed cgroup to avoid confusion.

[0048] Reference Figure 1 , shows a flowchart of a method for troubleshooting large-scale distributed services of the present application, wherein the method may specifically include the following steps:

[0049] In step S101, the target service is parsed;

[0050] In step S102, a control method based on a kubernetes application software custom resource definition extension mechanism is used, wherein the control method adopts a master-slave architecture, wherein the master-slave architecture includes a management node and a plurality of slave nodes, wherein the management node is used to coordinate the working status of the slave nodes, and a networking information environment variable is injected into each of the slave nodes to run the control method;

[0051] In step S103, based on the inter-process message passing tool, the management node issues commands, and collects and aggregates the results returned by all the slave nodes, so as to reduce the operations on the multiple target services to the operations on one target service;

[0052] In step S104, a full mesh link is constructed based on the kubernetes application software and the inter-process messaging tool;

[0053] In step S105, linking to the target service based on the nsenter mechanism;

[0054] In step S106, based on the freezing capability of the control group mechanism, the running state of the target service is frozen so as to troubleshoot the fault of the target service;

[0055] In step S107, after the troubleshooting is completed, the operating state of the target service is unfrozen and restored;

[0056] In step S108, the control method is deleted.

[0057] Furthermore, the target service is parsed, and the parsed information includes architecture type, number of container replicas, target node name, container name and container ID.

[0058] Furthermore, a control method based on the custom resource definition extension mechanism of kubernetes application software adopts a master-slave architecture and headless service. The master-slave architecture includes a management node (master) and a slave node (slave). The management node coordinates the working status of the slave node and injects the target service container ID, process ID, service name and port number of the management node into each slave environment variable.

[0059] Furthermore, based on the master-slave architecture, the management container and the slave container are run, and the slave container and the target service are run on the same node and have the same number of copies. The main purpose of doing this is:

[0060] First, low-latency communication.

[0061] When the slave container runs on the same node as the target service, the communication between them can be carried out through the local network interface, which greatly reduces the network latency. This is especially important for distributed systems that require frequent, real-time communication.

[0062] Local communication typically uses less bandwidth than cross-node communication because the packets do not need to pass through intermediate devices such as network switches and routers.

[0063] Second, resource optimization.

[0064] Containers running on the same node can share physical resources (such as CPU, memory, disk I / O) more efficiently, which helps reduce resource waste.

[0065] By deploying the slave containers and the target service on the same node, it is easier to enforce resource limits and quotas, ensuring that they do not interfere with each other while fully utilizing the node's resources.

[0066] Third, fault isolation and recovery.

[0067] If the slave container and the target service are deployed on the same node, they will be affected at the same time when the node fails. However, this design is usually combined with automatic restart and failure recovery mechanisms to ensure fast recovery of services.

[0068] Failure recovery at the node level makes it easier to maintain consistency of data and service status because related service components are all on the same node.

[0069] Fourth, the number of copies matches.

[0070] When the number of slave container replicas is the same as the number of target service replicas, it can ensure that each target service instance has a corresponding slave container for monitoring or coordination, thereby achieving a more even load distribution.

[0071] If the number of slave containers is greater than the number of target services, resources may be idle; if it is less than the number of target services, some target services may not be adequately monitored or coordinated. Therefore, matching the number of replicas is key to achieving redundancy and fault tolerance.

[0072] Fifth, simplify deployment and management.

[0073] Deploying slave containers on the same node as the target service and matching the number of replicas can simplify deployment scripts and configuration management.

[0074] Containers running on the same node are more easily monitored and logged together, which helps diagnose problems quickly.

[0075] Furthermore, the management node issues commands and collects and aggregates the results returned by all the slave nodes, including:

[0076] Dump the log information, process information, network information, CPU and memory information, RDMA network card, GPU status and NPU status of the target container in the kubernetes application software.

[0077] In a distributed system, the management node can merge and summarize the operation results of multiple targets by collecting the return results of the slave nodes and performing reduction operations on these results. This helps to simplify the output results of the system and improve the readability and ease of use of the system.

[0078] Furthermore, the construction of a full mesh link based on the kubernetes application software and the inter-process messaging tool includes:

[0079] Based on the service discovery capability of the kubernetes application software and the distributed capability of the inter-process messaging tool, during each traversal process, one of the slave nodes is selected as the server, and the other slave nodes are selected as the client;

[0080] The traversal is not completed until the last subordinate node is reached.

[0081] Furthermore, the link to the target service based on the nsenter mechanism includes:

[0082] The runtime privilege privileged given to the slave container is true, and the network privileges hostPID, hostIPC, and hostNetwork are all true;

[0083] Directly link to the namespace where the target program is located;

[0084] Use the above method as the entrypoint command to directly link to the target container when accessing remotely.

[0085] Furthermore, the freezing capability based on the control group mechanism freezes the running state of the target service, including:

[0086] The freezer.state of the target process of the target service is set to FROZEN to facilitate observation of the running state of the internal process of the target service.

[0087] Further, after the troubleshooting is completed, unfreezing and restoring the operating state of the target service includes:

[0088] Set the freezer.state of the target process to THAWED;

[0089] Finally, when you need to exit, delete the entire control method.

[0090] This application is combined with the Kubernetes platform, and by utilizing Kubernetes' service discovery function, service orchestration, and rich software ecology (tools), a Kubernetes-based troubleshooting method can be easily built end-to-end.

[0091] In order to solve the problem of difficult fault location of multi-machine and multi-card stateful services in Kubernetes distributed scenarios, a control method based on custom resource definition is proposed. This control method is a master-slave architecture. The master, as a management node, has two functions:

[0092] Function 1. Commands can be sent in batches to each slave node, and the results returned by all slaves can be collected and aggregated.

[0093] Function 2: As a reverse proxy component, the client can directly attach to each target service through the master, and then further operate the target container.

[0094] For this solution, the overall implementation flow chart is as follows: Figure 1 and Figure 2 The specific implementation steps are as follows:

[0095] First, parse out information such as architecture type, number of container replicas (replication), target node name (target node), container name, and container ID from the target job.

[0096] Second, the custom resource definition control method adopts the master-slave architecture and headless service, and injects the target service container ID, process ID, service name and port number of the management node into the environment variables of each slave node. The management node mainly coordinates the working status of the slave nodes. Among them, the slave container must run on the same node as the target service, and the number of replicas (replication) must be the same. The location of the management container is relatively free and unconstrained.

[0097] Third, based on the inter-process messaging tool, the message transmission between the management container and all subordinate containers is realized. Based on the inter-process messaging tool, commands can be issued in batches, and the results returned by all subordinate nodes can be collected and aggregated, thereby reducing the operations of multiple targets to the operations of one target. Specific operations include: transferring the log information, process information, network information, CPU and memory information, RDMA network card, GPU status and NPU status of the target container;.

[0098] Fourth, since network problems are the most common and difficult failures in distributed scenarios, it is necessary to strengthen them separately. P2P links need to be built in all target services to check whether the full-mesh communication links between the entire distributed services are normal.

[0099] The entire full mesh communication link is constructed by using the service discovery capability of the Kubernetes application software and the distributed capability of the inter-process messaging tool. During each traversal, only one slave node is the server, and the other slave nodes are clients. The above process is repeated until the last slave node is reached. This is mainly to achieve the following purposes:

[0100] Resource utilization and load balancing: By setting only one slave node as the server for each traversal, the resources in the cluster can be fully utilized and a single node can be prevented from becoming a bottleneck. At the same time, this design also helps to achieve load balancing, ensuring that each node can handle an appropriate amount of requests.

[0101] Simplified communication paths: In a fully meshed link architecture, each node is directly connected to other nodes. By setting the server and client roles, the communication path can be simplified and unnecessary network overhead can be reduced.

[0102] Improve system flexibility: Dynamic switching between server and client roles makes the system more flexible. During different traversals, different slave nodes can take turns to serve as the server, thereby balancing the system load and improving fault tolerance.

[0103] Easy troubleshooting and recovery: If a slave node fails or does not work properly, it is easy to bypass the node by adjusting the server and client roles to ensure the normal operation of the system. At the same time, this design also helps to quickly locate the fault point and recover.

[0104] Fifth, to further facilitate troubleshooting, you can set the privileged privileged of the slave container to true, and the network privileges hostPID, hostIPC, and hostNetwork to true. At the same time, use the nsenter mechanism to directly link to the namespace where the target program is located, and use this method as the entrypoint command to facilitate remote access and directly link to the target container.

[0105] Sixth, when encountering difficult problems, you need to suspend the running status of the target container. Based on the freezing capability of the control group mechanism, you can freeze the running of the target process. The specific method is to set the freezer.state of the target process to FROZEN. Freezing the fault site is conducive to observing the running status of the internal process of the target service. At this time, the target service is in a suspended state and will not exit even if there is an abnormality.

[0106] Seventh, when exiting for the last time, you need to unfreeze and restore the running state of the target process. The method is to set the freezer.state of the target process to THAWED and delete the entire control method.

[0107] In practice, we have found that in order to train larger models and run more complex services, dynamic networking of multiple machines and multiple cards for joint training is usually required. In the process of dynamic networking, many architectures have been derived, such as the master-worker architecture, the Rendezvous architecture, the Parameter Server architecture, and the scheduler-worker architecture. Each architecture is a specific implementation of a distributed service, and with it comes the question of how to quickly resolve problems encountered by distributed services. On the one hand, due to the inherent complexity of the distributed service itself, it is difficult to locate a fault when one occurs. It is necessary to check the working status of each node, run logs, and test whether p2p communication is normal. This process is time-consuming and labor-intensive, and as the number of nodes increases, the difficulty of locating the fault also increases exponentially. This method can greatly improve the efficiency of problem solving. Therefore,

[0108] Fall into the Huiju platform.

[0109] The working principle of the present invention is as follows:

[0110] A control method based on a custom resource definition extension mechanism of kubernetes application software, the control method is a master-slave structure, the control method can issue commands in batches, and collect and aggregate all results, and can transfer log information, process information, network information, CPU and memory information, RDMA network cards, and GPU status and NPU status of multiple target containers at one time.

[0111] Based on the nsenter principle, a full mesh communication link is built in the target service without intrusion to check whether the communication link is normal; at the same time, it is directly linked to the namespace where the target program is located, which is convenient for remote access and directly links to the inside of the target container.

[0112] For difficult problems, the running status of the target service can be suspended (frozen) based on the freezing capability of the control group mechanism, which is conducive to troubleshooting the problem of the target service.

[0113] The technical solution provided by this application may have the following beneficial effects:

[0114] This application solves the problem of difficult fault location in a multi-machine, multi-card stateful distributed training scenario: by optimizing the command issuance and collection steps, the number of manual accesses is reduced, and the operation of multiple targets is reduced to the operation of one target. All subordinate containers can be seamlessly accessed in the management container without switching views, which improves access efficiency. At the same time, the service discovery capability of the kubernetes application software and the distributed capability of the inter-process messaging tool are used to build a health check for the entire full-mesh communication link. Further configure the slave as a privileged container, use the nsenter mechanism to directly link to the namespace where the target program is located, and use this method as the entrypoint command to facilitate remote access and directly link to the inside of the target container. Finally, when encountering difficult problems or abnormal containers, it is necessary to suspend the running status of the target container, which is conducive to troubleshooting the target service, which is also a very effective method for containers. Through these steps, the speed of troubleshooting in a multi-machine, multi-card stateful distributed training scenario can be accelerated.

[0115] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0116] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0117] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0118] Figure 3 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0119] Reference Figure 3 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0120] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0121] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0123] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0124] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0125] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0126] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0127] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0128] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0129] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0130] Figure 419 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0131] Reference Figure 4 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0132] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0133] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0135] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0136] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A troubleshooting method for large-scale distributed services, characterized in that: The method comprises: Analyze the target service; A control method based on a kubernetes application software custom resource definition extension mechanism, the control method adopts a master-slave architecture, the master-slave architecture includes a management node and a plurality of slave nodes, the management node is used to coordinate the working status of the slave nodes, inject networking information environment variables into each of the slave nodes, and run the control method; Based on an inter-process messaging tool, the management node issues commands, and collects and aggregates results returned by all the slave nodes, so as to reduce operations on multiple target services to operations on one target service; Building a full mesh link based on the kubernetes application software and the inter-process messaging tool; Based on the nsenter mechanism, link to the target service; Freeze the running state of the target service based on the freezing capability of the control group mechanism, so as to troubleshoot the failure of the target service; After troubleshooting is completed, unfreeze and restore the operating status of the target service; Delete the control method.

2. The troubleshooting method for large-scale distributed services according to claim 1, characterized in that: The target service is parsed, and the parsed information includes architecture type, number of container copies, target node name, container name and container ID.

3. The troubleshooting method for large-scale distributed services according to claim 1, characterized in that: The environment variables injected into the network in each of the slave nodes include the target service container ID, the process ID, the service name and the port number of the management node.

4. The troubleshooting method for large-scale distributed services according to claim 3, characterized in that: Based on the master-slave architecture, a management container and the slave container are run, and the slave container and the target service are run on the same node and have the same number of copies.

5. The troubleshooting method for large-scale distributed services according to claim 1, characterized in that: The management node issues commands and collects and aggregates the results returned by all the slave nodes, including: The log information, process information, network information, CPU and memory information, RDMA network card, GPU status and NPU status of the target container in the kubernetes application software are transferred out.

6. The troubleshooting method for large-scale distributed services according to claim 1, characterized in that: The method of constructing a full mesh link based on the kubernetes application software and the inter-process message passing tool includes: Based on the service discovery capability of the kubernetes application software and the distributed capability of the inter-process messaging tool, during each traversal process, one of the slave nodes is selected as the server, and the other slave nodes are selected as the client; The traversal is not completed until the last subordinate node is reached.

7. The troubleshooting method for large-scale distributed services according to claim 4, characterized in that: The target service is linked based on the nsenter mechanism, including: Granting the slave container runtime privileges is true, and network privileges are true; Directly link to the namespace where the target program is located; Use the above method as the entrypoint command to directly link to the target container during remote access.

8. The troubleshooting method for large-scale distributed services according to claim 1, characterized in that: The freezing capability of the control group mechanism, freezing the running state of the target service, includes: The freezer.state of the target process of the target service is set to FROZEN to facilitate observation of the running state of the internal process of the target service.

9. The method for troubleshooting large-scale distributed services according to claim 8, characterized in that: After the troubleshooting is completed, unfreeze and restore the running state of the target service, including: Set the target process's freezer.state to THAWED.

10. The troubleshooting method for large-scale distributed services according to claim 7, characterized in that: The network privileges include hostPID, hostIPC, and hostNetwork.

Images