Abnormal computing power detection and positioning method, device and equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence, in particular to an abnormal computing power detection and positioning method and device, equipment and a storage medium, and the method comprises the steps: dividing a target program into a plurality of program slices, respectively inserting a preset computing power monitoring function into each program slice, and carrying out the detection and positioning of the abnormal computing power through the computing power monitoring functions; and obtaining resource load data of each terminal in the process of running one program slice, thereby determining the computing power health degree corresponding to the target program. Therefore, the resource load data of each terminal in the corresponding time period can be obtained in real time when each program slice is operated, so that the computing power state of each terminal is inferred in time according to the fluctuation trend of the resource load data, and computing power monitoring from a resource closed-loop layer is realized.

Description

Background Art

[0002] With the continuous improvement of computing power and the increase in data volume, the application areas of artificial intelligence (AI) models with large-scale parameters and complex structures, namely AI big models, are gradually expanding. AI big models refer to deep neural network models with millions or billions of parameters. Relying on their ability to process large-scale data, they have made significant progress and achievements in natural language processing, image recognition, speech recognition and other fields.

[0003] The training of large AI models usually requires the support of high-performance computing resources, such as standard graphics processing units (GPUs) or cloud clusters. During the training of large AI models, the required GPU computing power will reach more than 10,000 cards. However, low computing power efficiency or functional errors will affect the real-time and accuracy of the calculation results. Therefore, it is particularly important to determine whether there is an abnormality in computing power and to locate the abnormal computing power in order to ensure the training efficiency and completion of large AI models.

[0004] Under the relevant technology, for the detection and location of abnormal computing power during the training of large AI models, the traditional solution is to directly obtain the training results of each training task at different training stages, judge whether the status of the computing power is normal through the feedback of the task status results, and then perform binary search for each training task to complete the location of the abnormal computing power.

[0005] However, when using the above method, judging the computing power status through training results can only feedback the abnormal results, and cannot diagnose abnormal points during program operation. In addition, the location of abnormal computing power is started only after the operation results are obtained, which reduces the real-time performance of abnormal computing power location, thereby increasing the resource consumption of computing power operation and maintenance, and also increasing the complexity of large AI model training. Summary of the invention

[0006] The present application provides a method, device, equipment and storage medium for detecting and locating abnormal computing power, so as to improve the detection efficiency and positioning efficiency of abnormal computing power during the training of large AI models.

[0007] In a first aspect, an embodiment of the present application provides a method for detecting and locating abnormal computing power, the method comprising:

[0008] A preset program decomposition method is adopted to obtain multiple program slices contained in the target program, and a preset computing power monitoring function is respectively inserted into the multiple program slices, wherein the program slice is a code fragment in the target program that meets the preset constraint condition, and the computing power monitoring function is used to collect resource load data of each terminal running the corresponding program slice;

[0009] According to the running logic of the target program, multiple program slices are run in sequence. Each time a program slice is run, the following operations are performed:

[0010] Call a computing power monitoring function in a program slice and obtain a parameter set required by the computing power monitoring function, where the parameter set includes at least identification information of each terminal;

[0011] Based on the computing power monitoring function, the resource load data of each terminal at multiple time points during the running of a program slice is continuously obtained according to the set time interval;

[0012] Based on the obtained resource load data, the computing power health corresponding to the target program is determined, and when the computing power health indicates the existence of abnormal computing power, the target terminal corresponding to the abnormal computing power is determined based on the resource load data.

[0013] In a second aspect, an embodiment of the present application further provides a device for detecting and locating abnormal computing power, the device comprising:

[0014] A decomposition module, used to obtain multiple program slices contained in the target program by adopting a preset program decomposition method, and insert a preset computing power monitoring function into each of the multiple program slices, wherein the program slice is a code fragment in the target program that meets the preset constraint condition, and the computing power monitoring function is used to collect resource load data of each terminal running the corresponding program slice;

[0015] The running module is used to run multiple program slices in sequence according to the running logic of the target program, wherein each time a program slice is run, the following operations are performed:

[0016] Call a computing power monitoring function in a program slice and obtain a parameter set required by the computing power monitoring function, where the parameter set includes at least identification information of each terminal;

[0017] Based on the computing power monitoring function, the resource load data of each terminal at multiple time points during the running of a program slice is continuously obtained according to the set time interval;

[0018] The detection module is used to determine the computing power health corresponding to the target program based on the obtained resource load data, and when the computing power health indicates the existence of abnormal computing power, determine the target terminal corresponding to the abnormal computing power based on the resource load data.

[0019] In one possible implementation, when obtaining the parameter set required by the computing power monitoring function, the running module is used to:

[0020] Obtain the time point when the computing power monitoring function is called based on the system time, and save the time point to the time field included in the parameter set corresponding to the computing power monitoring function;

[0021] The terminal cluster preset for the corresponding target program is obtained, and the logical address of each terminal in the terminal cluster is used as the identification information of each terminal and saved in the terminal identification field of the parameter set.

[0022] In a possible implementation, based on the computing power monitoring function, when continuously acquiring resource load data of each terminal at multiple time points during the process of running a program slice according to a set time interval, the operation module is used to:

[0023] Acquire address information of a preset resource load database, wherein the resource load database stores: resource load data of each terminal at each time point during the process of running a program slice;

[0024] Based on the address information, the resource load database is searched, and based on the identification information of each terminal, the resource load data of each terminal at corresponding multiple time points are read from the resource load database at set time intervals.

[0025] In a possible implementation, when determining the computing power health corresponding to the target program based on the obtained resource load data, the detection module is used to:

[0026] For the resource load data of each terminal at multiple time points, the following operations are performed respectively: the resource load data of each terminal corresponding to a time point is aggregated to obtain a resource load data aggregate value corresponding to a time point;

[0027] For the resource load data summary values ​​corresponding to the obtained multiple time points, the following operations are performed: if the resource load data summary value at the first time point, relative to the resource load data summary value at the previous time point adjacent to the first time point, decreases by more than a preset threshold value, then it is determined that the computing power health corresponding to the target program at the first time point is abnormal, and the first time point is marked as an abnormal point, where the first time point is any one of the multiple time points.

[0028] In a possible implementation, when determining the target terminal corresponding to the abnormal computing power based on the load data of each resource, the detection module is used to:

[0029] For the resource load data corresponding to each terminal at the abnormal point, the following operations are respectively performed: comparing the resource load data of each terminal corresponding to the abnormal point to determine at least one abnormal resource load data corresponding to the abnormal point;

[0030] Based on identification information of a terminal corresponding to at least one abnormal resource load data, at least one target terminal corresponding to the abnormal computing power is determined.

[0031] In a possible implementation, the resource load data includes at least any one or any combination of the following:

[0032] Graphics processor GPU utilization;

[0033] GPU displays memory usage;

[0034] Central Processing Unit CPU utilization;

[0035] Memory usage;

[0036] Network traffic rate;

[0037] Disk input and output rate.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the first aspects when executing the computer program.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any method described in the first aspect.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when called by a computer, enables the computer to execute the method described in the first aspect.

[0041] The beneficial effects of this application are as follows:

[0042] In the embodiment of the present application, a content multi-target prediction method, device, equipment and storage medium are proposed. The server adopts a preset program decomposition method to divide the target program into multiple program slices, and inserts a preset computing power monitoring function in each program slice. Through the computing power monitoring function, the resource load data of each terminal in the process of running a program slice is obtained, so as to determine the computing power health corresponding to the target program. In this way, by inserting the computing power monitoring function at each program slice, the resource load data of each terminal in the corresponding time period can be obtained in real time every time a program slice is run, so that the computing power status of each terminal can be timely inferred through the fluctuation trend of the resource load data, and the computing power monitoring from the resource closed-loop layer is realized, avoiding the positioning of abnormal computing power after obtaining the program running results, resulting in complex positioning and redundant processes. Moreover, when abnormal computing power is monitored, the corresponding computing power monitoring function can be used to quickly locate the target terminal and target program slice corresponding to the abnormal computing power, thereby improving the detection efficiency and positioning efficiency of abnormal computing power, and reducing the unnecessary losses caused by program running errors, thereby saving the resource consumption of computing power operation and maintenance, and improving the reliability of AI large model training.

[0043] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0045] Figure 2 A flow chart of a method for detecting and locating abnormal computing power provided in an embodiment of the present application;

[0046] Figure 3 A logical diagram of running program slices in an embodiment of the present application;

[0047] Figure 4 A flowchart of the steps of running a program slice in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the fields included in a computing power monitoring function in an embodiment of the present application;

[0049] Figure 6 A flow chart of a method for a server to obtain a parameter set required for a computing power monitoring function in an embodiment of the present application;

[0050] Figure 7 This is a flow chart of a server reading resource load data based on a computing power monitoring function in an embodiment of the present application;

[0051] Figure 8 This is a schematic diagram of a log file generated after the server reads the resource load database at a certain minute in an embodiment of the present application;

[0052] Fig. 9 A fluctuation trend diagram of a GPU utilization rate over a period of time in an embodiment of the present application;

[0053] Fig.10 A flowchart of performing computing power health detection on a program for large model training in an embodiment of the present application;

[0054] Fig.11 A schematic diagram of a scenario in which abnormal computing power is detected and located by a program slice locator in an embodiment of the present application;

[0055] Fig.12 This is a schematic diagram of the structure of a device for detecting and locating abnormal computing power in an embodiment of the present application;

[0056] Fig.13 A schematic diagram of a hardware structure of a computing device in an embodiment of the present application;

[0057] Fig.14 A schematic diagram of the hardware composition structure of another computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0059] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0060] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0061] (1) AI big model: refers to a large and complex neural network that needs to store more parameters to increase the depth and width of the model, thereby improving the model's performance. The parameters start from tens of billions, and a large amount of data is trained to produce high-quality prediction results.

[0062] (2) Graphics Processing Unit (GPU): also known as display core, visual processor, or display chip, it is a microprocessor that specializes in performing image and graphics-related calculations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones).

[0063] (3) Display memory: This is the dedicated memory of the graphics card. It is used to store rendering data that has been processed or is about to be extracted by the graphics card chip. Just like the computer's memory, the video memory is a component used to store the graphics information to be processed.

[0064] (4) Central Processing Unit (CPU): It is the computing and control core of the computer system and the final execution unit for information processing and program execution.

[0065] (5) Memory: Also known as internal memory and main memory, it is used to temporarily store the calculation data in the CPU and the data exchanged with external memory such as the hard disk. It is a bridge for communication between the external memory and the CPU. All programs in the computer run in the memory. The strength of the memory performance affects the overall performance of the computer.

[0066] (6) Computing power: refers to computing power, that is, the ability of various processors such as CPU and GPU to perform data operations per second based on computer servers, high-performance computing clusters, various smart terminals and other supporting devices.

[0067] The following is a brief introduction to the design concept of the embodiment of the present application:

[0068] Researchers have been committed to using neural networks to simulate human cognitive processes and try to make computers have human-like intelligence. At present, with the continuous development of computer technology, especially the emergence of GPUs, people have begun to train larger-scale neural network models, namely AI large models.

[0069] When training large AI models, due to the large number of training parameters and the high scale of data exchange during training, the required GPU computing power will reach more than 10,000 cards. Therefore, multiple terminals are usually required to jointly provide GPU computing power. In the process of realizing the operation of heterogeneous computing power of 10,000 cards through multiple terminals, how to monitor the health of large-scale computing power in real time, and how to quickly locate the abnormal terminal when an abnormality occurs, are urgent problems to be solved in the training of large AI models.

[0070] Under related technologies, for monitoring the health of computing power of large AI models and locating abnormal computing power, the traditional solution is to directly feedback whether the status of computing power is normal through the training results of training tasks. When abnormal computing power occurs, it is necessary to perform a binary search on the task layer to locate the target terminal and target task corresponding to the abnormal computing power.

[0071] However, when using the above method, abnormal computing power is located only after the training results are obtained. The overall process connects the task layer and the resource layer in series, which is time-consuming and reduces the efficiency of AI developers. In addition, by feedbacking the computing power status through task results, only abnormal training results can be fed back. During the task process, abnormalities in running program slices cannot be fed back and pre-located, which increases the complexity of large model training operations.

[0072] In view of this, in the embodiments of the present application, a training method, device, equipment and storage medium for a role recognition model are proposed. The server adopts a preset program decomposition method to obtain multiple program slices contained in the target program, and inserts preset computing power monitoring functions into the multiple program slices respectively. According to the running logic of the target program, the multiple program slices are run in sequence. Each time a program slice is run, the computing power monitoring function is used to continuously obtain the resource load data of each terminal at multiple time points in the process of running a program slice according to the set time interval; based on the obtained resource load data, the computing power health corresponding to the target program is determined, and when the computing power health indicates the presence of abnormal computing power, the target terminal corresponding to the abnormal computing power is determined based on the resource load data.

[0073] In this way, by inserting a computing power monitoring function at each program slice, each time a program slice is run, the resource load data of each terminal in the corresponding time period can be obtained in real time, so that the computing power status of each terminal can be timely inferred through the fluctuation trend of the resource load data, realizing computing power monitoring from the resource closed-loop layer. Moreover, when abnormal computing power is detected, the corresponding computing power monitoring function can be used to quickly locate the target terminal and target program slice corresponding to the abnormal computing power, thereby improving the detection efficiency and positioning efficiency of abnormal computing power, thereby saving resource consumption of computing power operation and maintenance, and improving the reliability of large AI model training.

[0074] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0075] See also Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application.

[0076] The application scenario includes terminal devices 110 (including terminal devices 1101, 1102 ... terminal devices 110n) and a server 120. The terminal devices 110 and the server 120 can communicate with each other through a communication network.

[0077] In an optional implementation, the communication network may be a wired network or a wireless network. Therefore, the terminal device 110 and the server 120 may be directly or indirectly connected via a wired or wireless communication. For example, the terminal device 110 may be indirectly connected to the server 120 via a wireless access point, or the terminal device 110 may be directly connected to the server 120 via the Internet, and this application does not limit this.

[0078] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; various clients may be installed on the terminal device, and the client may be an application program (such as a browser, game software, etc.) that supports functions such as video preview and video playback, or a web page, a small program, etc.;

[0079] The server 120 is a background server corresponding to the client installed in the terminal device 110. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0080] It should be noted that the abnormal computing power detection and location method in the embodiment of the present application can be deployed in a computing device, which can be a server or a terminal device, wherein the server can be Figure 1 The server 120 shown in FIG. 1 may also be Figure 1 Other servers other than the server 120 shown in FIG. 1 ; the terminal device may be Figure 1 The terminal device 110 shown in FIG. 1 may also be Figure 1 Other terminal devices other than the terminal device 110 shown in the figure, that is, the method can be executed by a server or a terminal device alone, or can be executed by a server and a terminal device together.

[0081] It should be noted that, in the following, the example of execution by a server alone is mainly used for illustration, and no specific limitation is made here.

[0082] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices 110 and servers 120 is not limited and is not specifically limited in the embodiments of the present application.

[0083] In an embodiment of the present application, when there are multiple servers 120, multiple servers 120 can be combined into a blockchain, and the servers 120 are nodes on the blockchain; as disclosed in the embodiment of the present application, the abnormal computing power detection and positioning method, the multi-target prediction processing method involved therein, etc. can be saved on the blockchain.

[0084] See also Figure 2 As shown, it is a flow chart of a method for detecting and locating abnormal computing power provided in an embodiment of the present application. Figure 2, and explain in detail the specific operations performed:

[0085] Step S201: adopt a preset program decomposition method to obtain multiple program slices contained in the target program, and insert preset computing power monitoring functions into the multiple program slices respectively.

[0086] Among them, the program slice is a code fragment in the target program that meets the preset constraints, and the computing power monitoring function is used to collect resource load data of each terminal running the corresponding program slice.

[0087] Specifically, in an embodiment of the present application, in order to be able to monitor in real time the resource load status of each terminal during the execution of each part of the target program, so as to promptly discover abnormal computing power and stop the operation of the target program in time when abnormal computing power is detected, thereby reducing the losses caused by data calculation errors, the server first divides the target program into multiple program slices, and inserts a program slice locator into each program slice for real-time monitoring of the resource load data of each terminal during the operation of each program slice.

[0088] It should be noted that the program slice locator in this application, namely the computing power monitoring function, is a read-only insert statement for the target program of the large model, which has no effect on the overall operation of the target program. Therefore, it can be inserted at the head, middle or tail of the program slice, and this application does not impose any restrictions on this.

[0089] In addition, a program slice may contain one logical operation or multiple logical operations, and the code volume of any two program slices may be different. That is, the present application does not limit the specific method of dividing the target program into multiple program slices.

[0090] Step S202: Run multiple program slices in sequence according to the running logic of the target program.

[0091] Specifically, in the embodiment of the present application, when the server runs each program slice, multiple terminals need to jointly provide computing power, see Figure 3 As shown, it is a logical schematic diagram of running program slices in an embodiment of the present application. Each program slice runs on the server in sequence according to the running logic of the target program, and a terminal cluster composed of terminal 1, terminal 2, terminal 3, and terminal n jointly provides data computing capabilities for the running of the program slices. Among them, each terminal can process the same data at the same time, or process different data separately, and each terminal will communicate based on the communication network at regular intervals, exchange data with each other, and update parameters, etc.

[0092] For details, see Figure 4 As shown, it is a flowchart of the steps of running a program slice in an embodiment of the present application, wherein each time a program slice is run, the server performs the following operations:

[0093] Step S2021: Call a computing power monitoring function in a program slice and obtain the parameter set required by the computing power monitoring function.

[0094] The parameter set at least includes identification information of each terminal.

[0095] In an embodiment of the present application, the computing power monitoring function can be inserted into any position in a program slice. When the code of the program slice runs to the position of the computing power monitoring function, the server automatically calls the computing power monitoring function and obtains the parameter set required for the computing power monitoring function based on the server system and preset data.

[0096] See also Figure 5 As shown, it is a schematic diagram of the fields included in a computing power monitoring function in an embodiment of the present application. Since the target program in the embodiment of the present application is the program required for training the AI ​​large model, the fields included in the computing power monitoring function are: time, model training task name, model communication method and IPlist.

[0097] Among them, the granularity of the time field is at the minute level, the model training task name is the name of the AI ​​large model customized by relevant technical personnel, the model communication methods include the gradual attenuation method of Allreduce and the full communication method of all2all, and IPList is the identification information of each terminal in the terminal cluster that provides computing power for the target program. Except for the time field, the remaining fields are preset data.

[0098] Based on the above fields, see Figure 6 As shown, it is a flow chart of a method for the server to obtain the parameter set required for the computing power monitoring function in an embodiment of the present application, which specifically includes the following steps:

[0099] Step S2021-1: Obtain the time point when the computing power monitoring function is called based on the system time, and save the time point to the time field contained in the parameter set corresponding to the computing power monitoring function.

[0100] Specifically, when the computing power monitoring function is called, the server will obtain the current system time and save it to the time field contained in the parameter set, such as 2022 / 6 / 1 / 12:15.

[0101] Step S2021 - 2 : obtaining a terminal cluster preset for the corresponding target program, and saving the logical address of each terminal in the terminal cluster as identification information of each terminal in the terminal identification field of the parameter set.

[0102] Specifically, the server obtains the IP address of each terminal in the terminal cluster, forms an IP list and saves it in the terminal identification field, which is the core field of the computing power monitoring function. Based on the IP address of each terminal, the server can monitor and obtain the operating status of each terminal in real time. When a terminal is abnormal, the abnormal terminal can be located by IP address.

[0103] Step S2022: Based on the computing power monitoring function, the resource load data of each terminal at multiple time points during the running of a program slice is continuously obtained according to the set time interval.

[0104] In an embodiment of the present application, each terminal in the terminal cluster is configured with a client program for collecting its own resource load data. After the client program on each terminal continuously monitors and obtains the corresponding resource load data, it will uniformly store each resource load data in a preset resource load database and classify it according to the IP identification of each terminal, thereby ensuring that the server can index and search the resource load data of each terminal based on the IP identification.

[0105] For details, see Figure 7 As shown, it is a flow chart of the server reading resource load data based on the computing power monitoring function in an embodiment of the present application, which specifically includes the following steps:

[0106] Step S2022-1: Obtain address information of a preset resource load database.

[0107] The resource load database stores resource load data of each terminal at each time point during the process of running a program slice.

[0108] Specifically, in an embodiment of the present application, a KV (Key-Value) type storage system is used to save the resource load data of each terminal at each time point. It is a non-relational database that stores key-value pairs. The value can be any data of indefinite length, which can reduce time and space overhead and effectively reduce the number of disk reads and writes.

[0109] Furthermore, in the embodiment of the present application, the resource load data of each terminal includes at least any one or any combination of the following:

[0110] 1) Graphics processor GPU utilization.

[0111] 2) GPU displays memory usage.

[0112] 3) Central Processing Unit (CPU) utilization.

[0113] 4) Memory usage.

[0114] 5) Network traffic rate.

[0115] 6) Disk input and output rate.

[0116] Therefore, in the resource load database, the key information is the IP identification of each terminal, and the value information is the above one or more resource load data corresponding to the corresponding terminal.

[0117] Optionally, in actual applications, the resource load data that can be monitored may also include network bandwidth, CPU temperature and other resource load data that require targeted attention, and this application does not impose any restrictions on this.

[0118] Step S2022-2: Based on the address information, search the resource load database, and based on the identification information of each terminal, read the resource load data of each terminal at the corresponding multiple time points in the resource load database according to the set time interval.

[0119] Specifically, after obtaining the address information of the preset resource load database, the server sequentially indexes the resource load data of each terminal at corresponding multiple time points in the resource load database at intervals of one minute based on the IP identification of each terminal.

[0120] For example, assuming that the program slice being executed by the server is program slice 1, the number of terminals in the terminal cluster is 3, and the corresponding IP identifiers are IP1, IP2, and IP3, the server calls the computing power monitoring function corresponding to program slice 1. Starting from the start time point of program slice 1, every minute, the resource load data corresponding to the key information IP1, IP2, and IP3 is read from the resource load database until the program slice 1 ends.

[0121] See also Figure 8 As shown, it is a schematic diagram of the log file generated by the server after reading the resource load database in a certain minute in an embodiment of the present application, wherein the resource load data of each terminal is classified according to the IP identifier, and the summary value of different types of resource load data of all terminals is calculated. The server reads the resource load database once every minute and generates a corresponding log file.

[0122] Step S203: Based on the obtained resource load data, determine the computing power health corresponding to the target program, and when the computing power health indicates the presence of abnormal computing power, determine the target terminal corresponding to the abnormal computing power based on the resource load data.

[0123] Specifically, in the embodiment of the present application, the server adopts the method described in step S2022-2 to obtain the resource load data of each terminal at multiple time points when running program slice 1, and then determines the computing power health during the running of program slice 1 based on each resource load data, that is, the computing power health corresponding to the target program. The specific method is as follows:

[0124] For the resource load data of each terminal at multiple time points, the following operations are performed respectively: the resource load data of each terminal corresponding to a time point is aggregated to obtain a resource load data aggregation value corresponding to a time point.

[0125] For example, combined with Figure 8 As shown, the summary value of resource load data corresponding to a time point is the sum of the resource load data of each terminal corresponding to IP1, IP2, IP3, ... IPn at the current time point, that is, ∑IP(1, n). The corresponding resource load data fluctuation trend graph can be drawn based on the summary values ​​of resource load data corresponding to multiple time points.

[0126] See also Fig. 9 As shown, it is a fluctuation trend graph of GPU utilization over a period of time in an embodiment of the present application, wherein the horizontal axis is time, the time interval is one minute, and the vertical axis is the summary value of resource load data of each terminal at each time point during the execution of program slice 1 obtained by the computing power monitoring function corresponding to program slice 1. Through this fluctuation trend graph, it can be determined whether there is any abnormality in the GPU utilization of each terminal during the execution of program slice 1, thereby reflecting the computing power health of the target program.

[0127] Specifically, based on Fig. 9 The method for determining whether the GPU utilization is abnormal is as follows:

[0128] For the resource load data summary values ​​corresponding to the obtained multiple time points, the following operations are performed: if the resource load data summary value at the first time point, relative to the resource load data summary value at the previous time point adjacent to the first time point, decreases by more than a preset threshold value, then it is determined that the computing power health corresponding to the target program at the first time point is abnormal, and the first time point is marked as an abnormal point, where the first time point is any one of the multiple time points.

[0129] For example, if the GPU utilization corresponding to 2022 / 8 / 14 / 06:05 is 98.35%, and the GPU utilization corresponding to the previous adjacent time point, that is, 2022 / 8 / 14 / 06:04, is 84.77%, the GOU utilization has dropped by 13.58%, a decrease of approximately 14%, exceeding the preset threshold of 10%. It is determined that the computing health of the target program at 2022 / 8 / 14 / 06:05 is abnormal, and the time point is marked as an abnormal point.

[0130] Normally, two fluctuation trend graphs are drawn for GPU utilization and GPU video memory usage to reflect the overall computing power status of each terminal. The same fluctuation trend graphs can also be drawn for all types of resource load data to facilitate more comprehensive detection and judgment. This application does not impose any restrictions on this.

[0131] In the embodiment of the present application, after an abnormal point is determined, the operation of subsequent program slices will be stopped immediately, and the subsequent operation of the target program will not be restarted until the abnormal terminal is maintained normal.

[0132] Further, in the embodiment of the present application, after determining the abnormal point, the abnormal terminal corresponding to the abnormal point is determined based on the resource load data corresponding to each terminal, and the method is as follows:

[0133] For the resource load data corresponding to each terminal at the abnormal point, the following operations are performed respectively: the resource load data of each terminal corresponding to the abnormal point are compared to determine at least one abnormal resource load data corresponding to the abnormal point.

[0134] Based on identification information of a terminal corresponding to at least one abnormal resource load data, at least one target terminal corresponding to the abnormal computing power is determined.

[0135] For example, each outlier point has a corresponding Figure 8 The log file shown can be used to clearly compare the differences between the resource load data corresponding to each terminal. When a terminal is abnormal, its corresponding resource load data will have an obvious gap compared to the resource load data corresponding to other terminals. Therefore, at least one target terminal corresponding to the abnormal computing power can be determined through the log file corresponding to the abnormal point.

[0136] Based on the above embodiment, a specific application scenario is used below to illustrate the process of detecting and locating abnormal computing power using a program slice locator in the embodiment of the present application. Fig.10 As shown, it is a flowchart of performing computing power health detection on a program for large model training in an embodiment of the present application, specifically including:

[0137] Step S1001: The large model training target program is started.

[0138] Step S1002: Determine whether the target program's initial portion introduces a program slice locator, if so, execute step S1003, otherwise, execute step S1004.

[0139] Specifically, during the startup phase of the target program for large model training, a program slice locator is introduced at the beginning of the target program. This allows the resource load data of each terminal to be obtained once before the target program runs, and to detect in advance whether the status of each terminal is normal, further avoiding unnecessary losses caused by abnormal computing power during program operation.

[0140] Step S1003: Load the program slice locator header file.

[0141] Step S1004: Start the execution of a single program slice.

[0142] Step S1005: Determine whether the currently running program slice has been introduced into the program slice locator, if so, execute step S1006, otherwise execute step S1007.

[0143] Step S1006: Exporting running resource load data.

[0144] Step S1007: Execute the target program normally.

[0145] Specifically, when the program slice locator is not inserted, all programs are executed in sequence, and abnormal computing power is detected and located according to existing related technologies.

[0146] Step S1008: Draw a resource load data fluctuation trend chart.

[0147] Specifically, through the resource load data fluctuation trend chart, determine whether there is abnormal computing power during the operation of the current program slice. If not, continue to execute subsequent program slices. If so, immediately stop the operation of all programs, locate the abnormal computing power, and perform maintenance operations on the abnormal terminal.

[0148] See also Fig.11 As shown, it is a schematic diagram of the scenario in which abnormal computing power detection and positioning is performed by a program slice locator in an embodiment of the present application. The library file stores relevant configuration information for running the program slice locator and other functional functions. By inserting the program slice locator at the end of program slice 1, the resource load data of each terminal during the operation of program slice 1 can be exported by calling the program slice locator and its corresponding parameter set, and a resource load data fluctuation trend chart can be drawn to detect the computing power health corresponding to the target program.

[0149] To summarize, in the embodiments of the present application, by decomposing the target program into multiple program slices and inserting the corresponding computing power monitoring function into each program slice, the resource load data of each terminal when executing each program slice is monitored, and the computing power health of each terminal can be obtained in time. When the computing power health is abnormal, the abnormal terminal can be quickly located, thereby realizing abnormal computing power detection in a closed loop at the resource layer, thereby reducing the manpower investment in resource computing power operation and maintenance, and improving the accuracy and training efficiency of large model training.

[0150] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0151] Based on the same technical concept, see Fig.12 As shown, the embodiment of the present application also provides a detection and positioning device 1200 for abnormal computing power, the device comprising:

[0152] The decomposition module 1201 is used to obtain multiple program slices contained in the target program by adopting a preset program decomposition method, and insert a preset computing power monitoring function into each of the multiple program slices, wherein the program slice is a code fragment in the target program that meets the preset constraint condition, and the computing power monitoring function is used to collect resource load data of each terminal running the corresponding program slice;

[0153] The running module 1202 is used to run multiple program slices in sequence according to the running logic of the target program, wherein each time a program slice is run, the following operations are performed:

[0154] Call a computing power monitoring function in a program slice and obtain a parameter set required by the computing power monitoring function, where the parameter set includes at least identification information of each terminal;

[0155] Based on the computing power monitoring function, the resource load data of each terminal at multiple time points during the running of a program slice is continuously obtained according to the set time interval;

[0156] The detection module 1203 is used to determine the computing power health corresponding to the target program based on the obtained resource load data, and when the computing power health indicates the existence of abnormal computing power, determine the target terminal corresponding to the abnormal computing power based on the resource load data.

[0157] In a possible implementation, when obtaining the parameter set required by the computing power monitoring function, the operation module 1202 is used to:

[0158] Obtain the time point when the computing power monitoring function is called based on the system time, and save the time point to the time field included in the parameter set corresponding to the computing power monitoring function;

[0159] The terminal cluster preset for the corresponding target program is obtained, and the logical address of each terminal in the terminal cluster is used as the identification information of each terminal and saved in the terminal identification field of the parameter set.

[0160] In a possible implementation, based on the computing power monitoring function, when continuously acquiring resource load data of each terminal at multiple time points during the process of running a program slice according to a set time interval, the operation module 1202 is used to:

[0161] Acquire address information of a preset resource load database, wherein the resource load database stores: resource load data of each terminal at each time point during the process of running a program slice;

[0162] Based on the address information, the resource load database is searched, and based on the identification information of each terminal, the resource load data of each terminal at corresponding multiple time points are read from the resource load database at set time intervals.

[0163] In a possible implementation, when determining the computing power health corresponding to the target program based on the obtained resource load data, the detection module 1203 is used to:

[0164] For the resource load data of each terminal at multiple time points, the following operations are performed respectively: the resource load data of each terminal corresponding to a time point is aggregated to obtain a resource load data aggregate value corresponding to a time point;

[0165] For the resource load data summary values ​​corresponding to the obtained multiple time points, the following operations are performed: if the resource load data summary value at the first time point, relative to the resource load data summary value at the previous time point adjacent to the first time point, decreases by more than a preset threshold value, then it is determined that the computing power health corresponding to the target program at the first time point is abnormal, and the first time point is marked as an abnormal point, where the first time point is any one of the multiple time points.

[0166] In a possible implementation, when determining the target terminal corresponding to the abnormal computing power based on the load data of each resource, the detection module 1203 is used to:

[0167] For the resource load data corresponding to each terminal at the abnormal point, the following operations are respectively performed: comparing the resource load data of each terminal corresponding to the abnormal point to determine at least one abnormal resource load data corresponding to the abnormal point;

[0168] Based on identification information of a terminal corresponding to at least one abnormal resource load data, at least one target terminal corresponding to the abnormal computing power is determined.

[0169] In a possible implementation, the resource load data includes at least any one or any combination of the following:

[0170] Graphics processor GPU utilization;

[0171] GPU displays memory usage;

[0172] Central Processing Unit CPU utilization;

[0173] Memory usage;

[0174] Network traffic rate;

[0175] Disk input and output rate.

[0176] After introducing the method and apparatus for detecting and locating abnormal computing power according to an exemplary embodiment of the present application, next, a computing device according to another exemplary embodiment of the present application is introduced.

[0177] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0178] Based on the same inventive concept as the above method embodiment, the present application embodiment further provides a computing device 1300. In one embodiment, the computing device may be a server, such as Figure 1 In this embodiment, the structure of the computing device is as follows: Fig.13 As shown, it may include at least a memory 1301 , a communication module 1303 , and at least one processor 1302 .

[0179] The memory 1301 is used to store computer programs executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0180] The memory 1301 may be a volatile memory, such as a random-access memory (RAM); the memory 1301 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1301 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1301 may be a combination of the above memories.

[0181] The processor 1302 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1302 is used to implement the above-mentioned abnormal computing power detection and location method when calling the computer program stored in the memory 1301.

[0182] The communication module 1303 is used to communicate with terminal devices and other servers.

[0183] The specific connection medium between the memory 1301, the communication module 1303 and the processor 1302 is not limited in the embodiment of the present application. Fig.13 In the embodiment, the memory 1301 and the processor 1302 are connected via a bus 1304. The bus 1304 is connected to the processor 1302 via a bus 1304. Fig.13 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1304 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.13 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0184] The memory 1301 stores a computer storage medium, which stores computer executable instructions, which are used to implement the abnormal computing power detection and location method of the embodiment of the present application. The processor 1302 is used to execute the above abnormal computing power detection and location method.

[0185] In another embodiment, the computing device may also be other computing devices, such as Figure 1 In this embodiment, the structure of the computing device can be as follows: Fig.14 As shown, it includes: a communication component 1410, a memory 1420, a display unit 1430, a camera 1440, a sensor 1450, an audio circuit 1460, a Bluetooth module 1470, a processor 1480 and other components.

[0186] The communication component 1410 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the object to send and receive information through the WiFi module.

[0187] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1420. The memory 1420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1420 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1420 can store an operating system and various application programs, and can also store a computer program for executing the detection and positioning method of abnormal computing power in the embodiment of the present application.

[0188] The display unit 1430 may also be used to display information input by the object or information provided to the object and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1430 may include a display screen 1432 disposed on the front of the terminal device 110. The display screen 1432 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1430 may be used to display the game simulation game interface, etc. in the embodiment of the present application.

[0189] The display unit 1430 can also be used to receive input digital or character information, and generate signal input related to the object setting and function control of the terminal device 110. Specifically, the display unit 1430 may include a touch screen 1431 arranged on the front of the terminal device 110, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0190] The touch screen 1431 can be covered on the display screen 1432, or the touch screen 1431 and the display screen 1432 can be integrated to realize the input and output functions of the terminal device 110, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1430 can display the application and the corresponding operation steps.

[0191] The camera 1440 can be used to capture static images, and the subject can publish the images taken by the camera 1440 through the application. The camera 1440 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the processor 1480 to convert it into a digital image signal.

[0192] The terminal device may further include at least one sensor 1450, such as an acceleration sensor 1451, a distance sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0193] The audio circuit 1460, the speaker 1461, and the microphone 1462 can provide an audio interface between the object and the terminal device 110. The audio circuit 1460 can transmit the electrical signal converted from the received audio data to the speaker 1461, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1460 and converted into audio data, and then the audio data is output to the communication component 1410 to be sent to, for example, another terminal device, or the audio data is output to the memory 1420 for further processing.

[0194] The Bluetooth module 1420 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1420 to exchange data.

[0195] The processor 1480 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1420 and calling data stored in the memory 1420. In some embodiments, the processor 1480 may include one or more processing units; the processor 1480 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1480. In the present application, the processor 1480 can run the operating system, application programs, user interface display and touch response, as well as the detection and positioning method of abnormal computing power in the embodiment of the present application. In addition, the processor 1480 is coupled to the display unit 1430.

[0196] In addition, it should be noted that in the specific implementation of this application, object data related to the detection and positioning of abnormal computing power is involved. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0197] In some possible implementations, various aspects of the abnormal computing power detection and location method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computing device, the computer program is used to enable the computing device to execute the steps of the abnormal computing power detection and location method according to various exemplary embodiments of the present application described above in this specification.

[0198] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0199] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0200] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0201] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0202] The computer program for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the user computing device, partially on the user computing device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on the remote computing device. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0203] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0204] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0205] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting and locating abnormal computing power, characterized in that: include: A preset program decomposition method is adopted to obtain a plurality of program slices contained in a target program, and a preset computing power monitoring function is respectively inserted into the plurality of program slices, wherein the program slice is a code fragment in the target program that satisfies a preset constraint condition, and the computing power monitoring function is used to collect resource load data of each terminal running the corresponding program slice; According to the operation logic of the target program, the multiple program slices are run in sequence, wherein each time a program slice is run, the following operations are performed: Calling a computing power monitoring function in the one program slice, and obtaining a parameter set required by the computing power monitoring function, wherein the parameter set includes at least identification information of each terminal; Based on the computing power monitoring function, continuously obtaining resource load data of each terminal at multiple time points during the process of running the program slice according to a set time interval; Based on the obtained resource load data, the computing power health corresponding to the target program is determined, and when the computing power health indicates the existence of abnormal computing power, the target terminal corresponding to the abnormal computing power is determined based on the resource load data.

2. The method according to claim 1, characterized in that The obtaining of the parameter set required for the computing power monitoring function includes: Acquire the time point when the computing power monitoring function is called based on the system time, and save the time point to the time field included in the parameter set corresponding to the computing power monitoring function; A terminal cluster preset corresponding to the target program is obtained, and the logical address of each terminal in the terminal cluster is used as identification information of each terminal and saved in the terminal identification field of the parameter set.

3. The method according to claim 1, characterized in that The method of continuously acquiring resource load data of each terminal at multiple time points during the process of running the program slice based on the computing power monitoring function according to a set time interval includes: Acquire address information of a preset resource load database, wherein the resource load database stores: resource load data of each terminal at each time point during the process of running the program slice; Based on the address information, the resource load database is searched, and based on the identification information of each terminal, the resource load data of each terminal at the corresponding multiple time points are read in the resource load database according to the set time interval.

4. The method according to claim 1, 2 or 3, characterized in that: The determining the computing power health corresponding to the target program based on the obtained resource load data includes: For the resource load data of each terminal at multiple time points, the following operations are respectively performed: the resource load data of each terminal corresponding to a time point is aggregated to obtain a resource load data aggregation value corresponding to a time point; For the resource load data summary values ​​corresponding to the obtained multiple time points, the following operations are performed: if the resource load data summary value at the first time point, relative to the resource load data summary value at the previous time point adjacent to the first time point, decreases by more than a preset threshold value, then it is determined that the computing power health of the target program at the first time point is abnormal, and the first time point is marked as an abnormal point, wherein the first time point is any one of the multiple time points.

5. The method according to claim 4, characterized in that The determining, based on the resource load data, a target terminal corresponding to the abnormal computing power includes: For the resource load data corresponding to each terminal at the abnormal point, the following operations are respectively performed: comparing the resource load data of each terminal corresponding to the abnormal point to determine at least one abnormal resource load data corresponding to the abnormal point; Based on the identification information of the terminal corresponding to the at least one abnormal resource load data, at least one target terminal corresponding to the abnormal computing power is determined.

6. The method according to claim 1, 2 or 3, characterized in that: The resource load data includes at least one of the following or any combination thereof: Graphics processor GPU utilization; GPU displays memory usage; Central Processing Unit CPU utilization; Memory usage; Network traffic rate; Disk input and output rate.

7. A device for detecting and locating abnormal computing power, characterized in that: include: A decomposition module, used to obtain multiple program slices contained in the target program by adopting a preset program decomposition method, and insert a preset computing power monitoring function into each of the multiple program slices, wherein the program slice is a code fragment in the target program that meets the preset constraint condition, and the computing power monitoring function is used to collect resource load data of each terminal running the corresponding program slice; The running module is used to run the multiple program slices in sequence according to the running logic of the target program, wherein each time a program slice is run, the following operations are performed: Calling a computing power monitoring function in the one program slice, and obtaining a parameter set required by the computing power monitoring function, wherein the parameter set includes at least identification information of each terminal; Based on the computing power monitoring function, continuously obtaining resource load data of each terminal at multiple time points during the process of running the program slice according to a set time interval; The detection module is used to determine the computing power health corresponding to the target program based on the obtained resource load data, and when the computing power health indicates the existence of abnormal computing power, determine the target terminal corresponding to the abnormal computing power based on the resource load data.

8. A computing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that When the computer program product is called by a computer, the computer executes the method according to any one of claims 1 to 6.

Images