Data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the operation state data of a plurality of pieces of computing power equipment in a computing power equipment cluster at a current time node; performing state fluctuation analysis on the computing power equipment cluster to obtain a first fluctuation analysis result; under the condition that the first fluctuation analysis result represents that the fluctuation amplitude of the transmission state data of the current time node is greater than a first preset amplitude, determining a target processing data volume of each computing power device at the current time node; performing state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result; and determining an abnormal computing power device from the plurality of computing power devices based on the second fluctuation analysis result. According to the invention, the positioning efficiency and the positioning accuracy of the abnormal equipment can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] When training a large AI model, the number of parameters required for large model training is large and the amount of data exchanged during training is high, so the required computing power scale will reach more than 10,000 cards. Therefore, computing power equipment with large data processing capabilities is needed. For example, connecting multiple GPUs (Graphics Processing Units) to form a GPU cluster can meet the data processing needs of large data volumes. When running training tasks on such a large-scale computing power equipment, how to measure whether the computing power is running normally is an urgent problem to be solved in large model training.

[0003] In the prior art, the operating status of a computing device cluster can be measured based on the task running status. Although the success or failure status of a task can measure the operating status of a computing device cluster, it is impossible to locate abnormal devices in the computing device cluster when the computing device cluster is in an abnormal state. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a data processing method, device, electronic device and storage medium, which can obtain the status data of each computing device in the computing device cluster in real time, and then locate the abnormal device under abnormal conditions, improve the efficiency and convenience of computing device management, and improve the efficiency and accuracy of locating abnormal devices.

[0005] In order to solve the above technical problems, on the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Obtaining the operating status data of multiple computing devices in the computing device cluster at the current time node; the operating status data includes transmission status data characterizing the data transmission status of each computing device, and computing power status data characterizing the computing power usage status of each computing device;

[0007] Performing a state fluctuation analysis on the computing power device cluster based on the transmission state data of the multiple computing power devices at the current time node to obtain a first fluctuation analysis result;

[0008] When the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data of the historical time node is greater than the first preset amplitude, based on the transmission status data of each computing power device at the current time node and the computing power status data of each computing power device at the current time node, determine the target processing data volume of each computing power device at the current time node; the target processing data volume represents the data processing capacity of the corresponding computing power device;

[0009] Performing a state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result;

[0010] Based on the second fluctuation analysis result, an abnormal computing power device is determined from the multiple computing power devices; the fluctuation amplitude of the target processing data volume of the abnormal computing power device at the current time node relative to the target processing data volume of the abnormal computing power device at the historical time node is greater than a second preset amplitude.

[0011] On the other hand, an embodiment of the present application provides a data processing device, including:

[0012] An operation status data acquisition module is used to acquire the operation status data of multiple computing devices in the computing device cluster at the current time node; the operation status data includes transmission status data representing the data transmission status of each computing device, and computing power status data representing the computing power usage status of each computing device;

[0013] A first fluctuation analysis module, configured to perform state fluctuation analysis on the multiple computing power devices based on the transmission state data of the multiple computing power devices at the current time node to obtain a first fluctuation analysis result;

[0014] A first determination module is used to determine the target processing data volume of each computing power device at the current time node based on the transmission status data of each computing power device at the current time node and the computing power status data of each computing power device at the current time node, when the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data relative to the historical time node is greater than a first preset amplitude; the target processing data volume represents the data processing capacity of the corresponding computing power device;

[0015] A second fluctuation analysis module is used to perform a state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result;

[0016] The second determination module is used to determine an abnormal computing power device from the multiple computing power devices based on the second fluctuation analysis result; the fluctuation amplitude of the target processing data volume of the abnormal computing power device at the current time node relative to the target processing data volume of the abnormal computing power device at the historical time node is greater than a second preset amplitude.

[0017] On the other hand, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method as described above.

[0018] On the other hand, the present application provides a computer storage medium, wherein the storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by a processor and executes the data processing method as described above.

[0019] Implementing the embodiments of the present application has the following beneficial effects:

[0020] In this application, the transmission status data of the computing power device cluster at the current time node can be subjected to fluctuation analysis, and when the fluctuation amplitude of the transmission status data is greater than the first preset amplitude, the target processing data volume of each computing power device at the current time node can be further determined based on the transmission status data and computing power status data of each computing power device in the computing power device cluster; then, based on the fluctuation amplitude of the target processing data volume of each computing power device, the abnormal computing power device is determined from the computing power device cluster. In this application, by obtaining the transmission status data and computing power status data related to the performance of the computing power device, it is convenient to understand the performance status of each computing power device in the computing power device cluster, and improve the efficiency and convenience of computing power device management; further, when it is determined that the transmission status data of the computing power device cluster is abnormal, the abnormal computing power device is determined based on the fluctuation of the target processing data volume of each computing power device. Since the target processing data volume is calculated based on the real operating status data obtained in real time, the real-time and accuracy of the target processing data volume are guaranteed, and then the abnormal computing power device is determined based on the target processing data volume, and the positioning efficiency and positioning accuracy of the abnormal device are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of the implementation environment provided by the embodiment of the present application;

[0023] Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application;

[0024] Figure 3 This is a flow chart of a method for determining an abnormal computing power node provided in an embodiment of the present application;

[0025] Figure 4 It is a flow chart of a method for analyzing state fluctuations of a computing power device cluster provided in an embodiment of the present application;

[0026] Figure 5 is a flow chart of a method for determining a target processing data volume of a computing device provided in an embodiment of the present application;

[0027] Figure 6 It is a flow chart of a method for processing state data of a computing device including multiple computing nodes provided in an embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of the data processing architecture of the present application provided by an embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the operating status fluctuation of the computing power device cluster provided in the embodiment of the present application;

[0030] Fig. 9 This is a schematic diagram of fluctuations in target processing data volume of a force device provided in an embodiment of the present application;

[0031] Fig.10 It is a flow chart of the operating status observation method of the computing power device cluster provided in the embodiment of the present application;

[0032] Fig.11 is a schematic diagram of a data processing device provided in an embodiment of the present application;

[0033] Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] To make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] See also Figure 1 , which shows a schematic diagram of the implementation environment provided by an embodiment of the present application, the implementation environment may include: a computing power device cluster 110, a cluster management terminal 120 and a task submission terminal 130; the computing power device cluster 110 and the task submission terminal 130 can communicate through the network, the cluster management terminal 120 and the task submission terminal 130 can communicate through the network, and the cluster management terminal 120 and each computing power device in the computing power device cluster 110 can communicate.

[0037] Specifically, the task submission terminal 130 can submit the pending tasks to the computing power device cluster 110, and the computing power device cluster 110 can process the pending tasks based on the multiple computing power devices it contains, and return the task processing results to the task submission terminal 130. Among them, when the computing power device cluster 110 processes the pending tasks, the cluster management terminal 120 can collect and store the operating status data of each computing power device in the computing power device cluster; based on the operating status data collected at each collection time point, the computing power performance of the computing power device cluster is analyzed, and an alarm is issued when an abnormality occurs in the computing power device.

[0038] The task submission terminal 130 can communicate with the computing device cluster 110 based on the browser / server mode (B / S) or the client / server mode (C / S). The task submission terminal 130 may include: physical devices such as smart phones, tablet computers, laptops, digital assistants, smart wearable devices, vehicle terminals, servers, etc., and may also include software running in physical devices, such as applications, etc. The operating system running on the task submission terminal 130 in the embodiment of the present application may include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0039] The computing device cluster 110 and the task submission terminal 130 can establish a communication connection via wired or wireless communication. Each device in the computing device cluster 110 can be a physical server or a cloud server.

[0040] The cluster management terminal 120 can establish a communication connection with the computing power device cluster 110 via wired or wireless communication. The cluster management terminal 120 can include an independently running server, or a distributed server, or a server cluster composed of multiple servers, where the server can be a cloud server or a physical server.

[0041] In order to solve the problem in the prior art that it is impossible to locate abnormal devices in a computing power device cluster when the computing power device cluster is in an abnormal state, the embodiment of the present application provides a data processing method, the execution subject of which can be the above-mentioned cluster management end; please refer to Figure 2 , the method may include:

[0042] S210. Obtain the operating status data of multiple computing devices in the computing device cluster at the current time node; the operating status data includes transmission status data characterizing the data transmission status of each computing device, and computing power status data characterizing the computing power usage status of each computing device.

[0043] In this embodiment, the computing power device cluster includes multiple computing power devices, each of which has data processing capabilities. The data processing capabilities of different computing power devices can be the same or different; each computing power device can exchange data with at least one other computing power device in the computing power device cluster. Specifically, the computing power device can be suitable for a large number of high-density calculations with simple logic but high concurrency. The computing power device can specifically be a device including a GPU computing power unit, or a device with the same or similar characteristics as a GPU computing power unit. By obtaining the operating status data of multiple computing power devices in the computing power device cluster, the operating status of multiple computing power devices can be analyzed, and in the case of abnormal decomposition results, fault warnings can be issued in a timely manner to ensure the smooth progress of computing tasks in the computing power device cluster.

[0044] Each computing device in the computing device cluster may have a data acquisition interface, so that the cluster management end may obtain the transmission status data and computing power status data of each computing device through the data acquisition interface. The cluster management end may obtain the real-time transmission status data and real-time computing power status data of each computing device at the current data acquisition node in real time through the data acquisition interface of each computing device at each data acquisition node. The transmission status data can characterize the data transmission status of each computing device, and may include the data transmission status of each computing device and other computing devices in the computing device cluster, as well as the data transmission status between each functional module within each computing device. The computing power status data can characterize the computing power usage of each computing device. By obtaining the computing power status data of each computing device, the computing power resources used and the idle computing power resources of each computing device can be determined. In a specific embodiment, the computing tasks can be allocated among multiple computing devices based on the computing power resources used and the idle computing power resources of each computing device, so as to reallocate the tasks on the computing devices with larger loads to the computing devices with smaller loads, which can reduce the data processing pressure of the computing devices with larger loads and improve the task processing efficiency.

[0045] S220. Perform a state fluctuation analysis on the computing power device cluster based on the transmission state data of the multiple computing power devices at the current time node to obtain a first fluctuation analysis result.

[0046] In this embodiment, the state fluctuation analysis of the computing power device cluster refers to the fluctuation analysis of the overall operating state of the computing power device cluster. Specifically, the state fluctuation analysis can be based on the transmission state data of the computing power device cluster at the current time node and the transmission state data of the computing power device cluster at the historical time node. The historical time node here can be the previous time node of the current time node, and the historical time node can also be multiple historical time nodes located before the current time node.

[0047] In the case where the historical time node is the previous time node, a state fluctuation analysis can be performed based on the transmission state data of the computing power device cluster at the current time node and the transmission state data at the previous time node to obtain a first fluctuation analysis result. Specifically, the sum of the transmission state data of multiple computing power devices at the current time node can be compared with the sum of the transmission state data of multiple computing power devices at the previous time node to obtain the first fluctuation analysis result. Accordingly, the first difference between the sum of the transmission state data of multiple computing power devices at the current time node and the sum of the transmission state data of multiple computing power devices at the previous time node can be calculated, and the fluctuation amplitude can be determined based on the absolute value of the first difference. In an example, a first average value can also be calculated based on the sum of the transmission status data of the computing power device cluster at the current time node and the number of computing power devices in the computing power device cluster at the current time node; a second average value can be calculated based on the sum of the transmission status data of the computing power device cluster at the previous time node and the number of computing power devices in the computing power device cluster at the previous time node; a second difference between the first average value and the second average value is calculated, and the fluctuation amplitude is determined based on the absolute value of the second difference; this method can be applied to scenarios where the number of computing power devices at the current time node is inconsistent with that at the previous time node, that is, the number of computing power devices at the current time node has changed relative to the previous time node. If the method of determining the fluctuation analysis based on the sum of the transmission status data is still used when the number of computing power devices has changed, analysis errors will result. The average transmission status data of a single computing power device can be reflected by averaging the sum of the transmission status data, thereby avoiding the impact of changes in the number of computing power devices on the fluctuation analysis.

[0048] In the case where the historical time nodes are multiple historical time nodes located before the current time node, the sum of the transmission status data of the multiple historical time nodes can be calculated, and based on the sum of the transmission status data of the multiple historical time nodes and the number of computing devices in the computing device cluster (when the number of computing devices in the computing device cluster remains unchanged), a third average value is calculated, and then the third difference between the transmission status data of the current time node and the third average value is calculated, and the fluctuation range is determined based on the absolute value of the third difference. By determining the fluctuation of the transmission status data of the current time node relative to the historical time node by taking the average value of the sum of the transmission status data of multiple historical time nodes, the stability and accuracy of the fluctuation analysis can be improved.

[0049] In this embodiment, determining the fluctuation amplitude based on the absolute value of the difference may be to directly determine the difference as the fluctuation amplitude, or to determine the fluctuation amplitude by the ratio of the absolute value of the difference to the transmission status data of the historical time node, or to determine the fluctuation amplitude by the ratio of the absolute value of the difference to the average value of the transmission status data of the historical time node. Accordingly, the first fluctuation analysis result may specifically be the fluctuation amplitude determined based on the difference, or the fluctuation amplitude determined based on the ratio of the difference to the transmission status data of the historical time node, or the fluctuation amplitude determined based on the ratio of the absolute value of the difference to the average value of the transmission status data of the historical time node. In one example, the first fluctuation analysis result may also be the comparison result of the actual fluctuation amplitude of the transmission status data and the first preset fluctuation amplitude.

[0050] S230. When the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data relative to the historical time node is greater than the first preset amplitude, based on the transmission status data of each computing power device at the current time node and the computing power status data of each computing power device at the current time node, determine the target processing data volume of each computing power device at the current time node; the target processing data volume represents the data processing capability of the corresponding computing power device.

[0051] According to the above content, the first preset amplitude here can be a preset difference value of the transmission status data, or can be a percentage, which can be determined according to the specific implementation situation.

[0052] When the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data relative to the historical time node is greater than the first preset amplitude, it indicates that abnormal fluctuation occurs in the computing power device cluster, that is, abnormal computing power devices appear in the computing power device cluster. In order to further locate the abnormal computing power devices, the target processing data volume of each computing power device at the current time node can be determined based on the operating status data of each computing power device at the current time node. The target processing data volume can characterize the data processing capability of the computing power device.

[0053] When the first fluctuation analysis result indicates that the fluctuation amplitude of the transmission status data of the current time node relative to the transmission status data of the historical time node is less than or equal to the first preset amplitude, it means that the computing power device cluster is operating normally and there is no abnormal computing power device, which means that there is no need to perform subsequent operations to locate the abnormal computing power device.

[0054] S240. Perform a state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result.

[0055] The state fluctuation analysis of each computing power device can be achieved based on the target processing data volume of the computing power device at the corresponding time node. Specifically, the state fluctuation analysis can be based on the target processing data volume of each computing power device at the current time node and the target processing data volume of the computing power device at the historical time node. The historical time node here can be the previous time node of the current time node, and the historical time node can also be multiple historical time nodes located before the current time node.

[0056] When the historical time node is the previous time node, a state fluctuation analysis can be performed based on the target processing data volume of each computing power device at the current time node and the target processing data volume of the previous time node to obtain a second fluctuation analysis result. Specifically, the fourth difference between the target processing data volume of each computing power device at the current time node and the target processing data volume of the previous time node can be calculated, and the fluctuation amplitude can be determined based on the absolute value of the fourth difference.

[0057] In the case where the historical time nodes are multiple historical time nodes located before the current time node, the sum of the target processing data volume of each computing power device at the multiple historical time nodes can be calculated, and based on the sum of the target processing data volume of each computing power device at the multiple historical time nodes and the number of computing power devices in the computing power device cluster (when the number of computing power devices in the computing power device cluster remains unchanged), a fourth average value is calculated, and then a fifth difference between the target processing data volume of each computing power device at the current time node and the fourth average value is calculated, and the fluctuation range is determined based on the absolute value of the fifth difference. By determining the fluctuation of the target processing data volume of the current time node relative to the historical time node by taking the average value of the sum of the target processing data volume of each computing power device at multiple historical time nodes, the stability and accuracy of the fluctuation analysis can be improved.

[0058] In this embodiment, determining the fluctuation amplitude based on the absolute value of the difference may be directly determining the difference as the fluctuation amplitude, or determining the fluctuation amplitude by the ratio of the absolute value of the difference to the target processing data volume of the historical time node, or determining the fluctuation amplitude by the ratio of the absolute value of the difference to the average value of the target processing data volume of the historical time node. Accordingly, the second fluctuation analysis result may specifically be the fluctuation amplitude determined based on the difference, or the fluctuation amplitude determined based on the ratio of the difference to the target processing data volume of the historical time node, or the fluctuation amplitude determined based on the ratio of the absolute value of the difference to the average value of the target processing data volume of the historical time node. In one example, the second fluctuation analysis result may also be the comparison result of the actual fluctuation amplitude of the target processing data volume and the second preset fluctuation amplitude.

[0059] S250. Based on the second fluctuation analysis result, an abnormal computing power device is determined from the multiple computing power devices; the fluctuation amplitude of the target processing data volume of the abnormal computing power device at the current time node relative to the target processing data volume of the abnormal computing power device at the historical time node is greater than a second preset amplitude.

[0060] According to the above content, it can be known that the second fluctuation analysis result can be a fluctuation amplitude determined based on the difference, or a fluctuation amplitude determined based on the ratio of the difference to the target processing data volume at the historical time node, or a fluctuation amplitude determined based on the ratio of the absolute value of the difference to the average value of the target processing data volume at the historical time node. In one example, the second fluctuation analysis result can also be a comparison result of the actual fluctuation amplitude of the target processing data volume with the second preset fluctuation amplitude; thereby, based on the second fluctuation analysis result, the abnormal computing power device with abnormal fluctuation of the target processing data volume can be located from multiple computing power devices.

[0061] In this application, the transmission status data of the computing power device cluster at the current time node can be subjected to fluctuation analysis, and when the fluctuation amplitude of the transmission status data is greater than the first preset amplitude, the target processing data volume of each computing power device at the current time node can be further determined based on the transmission status data and computing power status data of each computing power device in the computing power device cluster; then, based on the fluctuation amplitude of the target processing data volume of each computing power device, the abnormal computing power device is determined from the computing power device cluster. In this application, by obtaining the transmission status data and computing power status data related to the performance of the computing power device, it is convenient to understand the performance status of each computing power device in the computing power device cluster, and improve the efficiency and convenience of computing power device management; further, when it is determined that the transmission status data of the computing power device cluster is abnormal, the abnormal computing power device is determined based on the fluctuation of the target processing data volume of each computing power device. Since the target processing data volume is calculated based on the real operating status data obtained in real time, the real-time and accuracy of the target processing data volume are guaranteed, and then the abnormal computing power device is determined based on the target processing data volume, and the positioning efficiency and positioning accuracy of the abnormal device are improved.

[0062] In this embodiment, each computing device includes multiple computing nodes, and the transmission status data of each computing device includes the transmission status data corresponding to each of the multiple computing nodes, and the computing status data of each computing device includes the computing status data corresponding to each of the multiple computing nodes; thus, after determining the abnormal computing device, the abnormal computing node in the abnormal computing device can be further determined based on the operating status data of the multiple computing nodes in the abnormal computing device; for details, please refer to Figure 3 , which shows a method for determining abnormal computing power nodes, which may include:

[0063] S310. Perform averaging processing based on the transmission status data corresponding to each of the multiple computing power nodes to obtain average transmission status data.

[0064] S320. Perform averaging processing based on the computing power status data corresponding to each of the multiple computing power nodes to obtain average computing power status data.

[0065] The computing nodes of each computing power device can perform data parallel processing operations. Each computing power node has corresponding transmission status data and computing power status data. By averaging the transmission status data and computing power status data of multiple computing power nodes in each computing power device, the average transmission status data and average computing power status data of multiple computing power nodes in each computing power device can be obtained. The average transmission status data can represent the average level of data transmission of multiple computing power nodes, and the average computing power status data can represent the average level of computing power status of multiple computing power nodes.

[0066] S330. Compare the transmission status data of each computing power node with the average transmission status data to obtain a first comparison result.

[0067] S340. Compare the computing power status data of each computing power node with the average computing power status data to obtain a second comparison result.

[0068] Compare the actual transmission status data of each computing power node with the average transmission status data to obtain a first comparison result, which can indicate the size relationship between the actual transmission status data of each computing power node and the average transmission status data. Compare the actual computing power status data of each computing power node with the average computing power status data to obtain a second comparison result, which can indicate the size relationship between the actual computing power status data of each computing power node and the average computing power status data.

[0069] S350. Based on the first comparison result and the second comparison result, an abnormal computing power node is determined from the multiple computing power nodes of the abnormal device; the first comparison result of the abnormal computing power node indicates that the transmission status data of the abnormal computing power node is smaller than the average transmission status data, and / or the second comparison result of the abnormal computing power node indicates that the computing power status data of the abnormal computing power node is smaller than the average computing power status data.

[0070] The abnormal computing power node in the abnormal device can be determined through the first comparison result and the second comparison result; the abnormal computing power node can be a computing power node whose transmission status data is less than the average transmission status data, or a computing power node whose computing power status data is less than the average computing power status data, or a computing power node whose transmission status data is less than the average transmission status data and whose computing power status data is less than the average computing power status data.

[0071] After determining the abnormal computing power device, it is possible to further determine the abnormal computing power nodes in the abnormal computing power device based on the operating status data of multiple computing power nodes in the abnormal computing power device, thereby improving the positioning accuracy of the abnormal computing power nodes, thereby achieving fine-grained abnormality positioning, facilitating subsequent abnormality repair, and improving abnormality repair efficiency.

[0072] In this embodiment, the transmission status data of each computing device includes a first data transmission rate between each computing device and other computing devices, and a second data transmission rate between the video memory and the internal memory in each computing device, wherein the other computing devices are computing devices other than the current computing device among the multiple computing devices; the first data transmission rate refers to the data transmission rate between each computing device and other computing devices, for example, it may specifically refer to the network throughput of each computing device; the second data transmission rate refers to the data transmission rate between the video memory and the internal memory in each computing device, for example, the highest data transmission rate between the video memory and the internal memory in each computing device is the bus bandwidth between the internal memory and the video memory. Therefore, when analyzing the state fluctuations of the computing device cluster, a comprehensive analysis can be performed based on the first data transmission rate and the second data transmission rate. For details, please refer to Figure 4 , which shows a method for analyzing state fluctuations of a computing power device cluster, which may include:

[0073] S410. Based on the sum of the first data transmission rate of each computing power device at the current time node and the second data transmission rate of each computing power device at the current time node, the data transmission rate of each computing power device at the current time node is obtained.

[0074] In this embodiment, since the first data transmission rate and the second data transmission rate can characterize the data transmission capabilities of different dimensions, that is, both can characterize the data transmission capabilities of the computing power device, the two data transmission rates can be summed, and the data transmission rate after the sum can reflect the total data transmission rate of each computing power device; accordingly, the first data transmission rate and the second data transmission rate of each computing power device at the current time node are summed to obtain the data transmission rate of each computing power device at the current time node.

[0075] S420. Determine the sum of the data transmission rates of the multiple computing devices at the current time node as the total data transmission rate at the current time node.

[0076] Based on the data transmission rate of each computing device at the current time node, the data transmission rate of each computing device at the current time node can be further summed to obtain the total data transmission rate of multiple computing devices at the current time node.

[0077] S430. Determine the transmission rate fluctuation range based on the total data transmission rate at the current time node and the total data transmission rate at the historical time nodes.

[0078] The calculation method of the total data transmission rate of the historical time node is consistent with the total data transmission rate of the current time node, which will not be repeated here. For the specific calculation method of the transmission rate fluctuation range, please refer to the specific implementation details of step S220 above in this embodiment, that is, the historical time node can be the previous time node of the current time node, and the historical time node can also be multiple historical time nodes located before the current time node.

[0079] When the historical time node is the previous time node, the sixth difference between the total data transmission rate of the current time node and the total data transmission rate of the historical time node can be calculated, and the transmission rate fluctuation range can be determined based on the absolute value of the sixth difference.

[0080] When the historical time nodes are multiple historical time nodes located before the current time node, the sum of the total data transmission rates of the multiple historical time nodes can be calculated, and based on the sum of the total data transmission rates of the multiple historical time nodes and the number of computing power devices in the computing power device cluster, the fifth average value is calculated, and then the seventh difference between the total data transmission rate of the current time node and the fifth average value is calculated, and the transmission rate fluctuation amplitude is determined based on the absolute value of the seventh difference.

[0081] The transmission rate fluctuation amplitude in this embodiment can be directly determined by the absolute value of the difference as the transmission rate fluctuation amplitude, or it can be determined by the ratio of the absolute value of the difference to the total data transmission rate of the historical time node as the transmission rate fluctuation amplitude, or it can be determined by the ratio of the absolute value of the difference to the average value of the total data transmission rate of the historical time node as the transmission rate fluctuation amplitude.

[0082] S440. Determine the first fluctuation analysis result based on the transmission rate fluctuation amplitude.

[0083] Correspondingly, the first fluctuation analysis result may include the above-mentioned transmission rate fluctuation amplitude. The fluctuation of the data transmission rate at the current time node may be known through the first fluctuation analysis result, thereby facilitating the analysis of the operating status of the computing power equipment cluster based on the first fluctuation analysis result.

[0084] Specifically, the first data transmission rate of each computing device is recorded as A, and the second data transmission rate is recorded as B, so that the calculation formula of the total data transmission rate M of multiple computing devices in the computing device cluster is shown in formula (1):

[0085] M=∑(A i +B i ) (1)

[0086] Where i refers to the i-th computing device, and 0≤i≤number of computing devices-1, (A i +B i ) is the sum of the first data transmission rate and the second data transmission rate of the i-th computing power device. If the fluctuation range of M relative to the historical time node is less than or equal to the first preset range, which may be 5%, a%, etc., then it is determined that the operation status of the computing power device cluster is normal; if the fluctuation range of M relative to the historical time node is greater than the first preset range, then it is determined that the operation status of the computing power device cluster is abnormal, and further analysis is required to determine the abnormal computing power device.

[0087] In this embodiment, the data transmission capability of each computing power device is characterized by the first data transmission rate between each computing power device and other computing power devices, and the second data transmission rate between the video memory and the internal memory in each computing power device. Since the first data transmission rate and the second data transmission rate can characterize data transmission capabilities in different dimensions, the comprehensiveness of the characterization of the data transmission rate of the computing power device is improved, and the transmission rate fluctuation analysis is performed based on the comprehensive data transmission rate, which can improve the accuracy of the transmission rate fluctuation analysis and thus improve the accuracy of the analysis of the operating status of the computing power device cluster.

[0088] Specifically, the computing power status data of each computing power device includes the video memory usage of each computing power device and the computing power resource utilization rate of each computing power device; thus, the amount of data that each computing power device can process per unit of video memory and per unit of computing power can be determined based on the video memory usage and computing power resource utilization rate of each computing power device. Each computing power device has video memory space, and the video memory usage of each computing power device is the size of the video memory space used by each computing power device; the resources that can be used to perform computing power operations in each computing power device are the total computing power resources, and the computing power resource utilization rate can be the ratio of the used computing power resources to the total computing power resources. For details, please refer to Figure 5 , which shows a method for determining a target processing data volume of a computing device, the method may include:

[0089] S510. Perform a product operation based on the video memory usage of each computing device at the current time node and the computing resource utilization rate of each computing device at the current time node to obtain the product operation result of each computing device.

[0090] S520. Based on the data transmission rate of each computing power device at the current time node and the ratio of the multiplication operation result, determine the target processing data volume of each computing power device at the current time node; the target processing data volume represents the amount of data that can be processed by unit video memory and unit computing power.

[0091] When determining the target processing data volume of each computing device at the current time node, the data transmission rate of each computing device at the current time node is used as the numerator, the product operation result of each computing device at the current time node is used as the denominator, and the corresponding ratio is determined as the target processing data volume of each computing device at the current time node. The target processing data volume can specifically be the throughput data of unit video memory and unit computing power processing.

[0092] The first data transmission rate of each computing device is recorded as A, the second data transmission rate is recorded as B, the video memory usage is recorded as C, and the computing resource utilization rate is recorded as D. The calculation formula for the target processing data volume N of each computing device at the current time node is shown in formula (2):

[0093]

[0094] The computing power status data of each computing power device may include the video memory usage and computing power resource utilization rate of each computing power device. The video memory usage and computing power resource utilization rate can characterize the status information of the computing power device from different dimensions. On this basis, combined with the data transmission rate, video memory usage and computing power resource utilization rate of each computing power device at the current time node, the amount of data that each computing power device can process per unit video memory and per unit computing power can be determined, and the performance of each computing power device is measured with unified unit data, which facilitates subsequent fluctuation analysis based on the target processing data volume of different computing power devices, thereby improving the convenience of fluctuation analysis.

[0095] In one example, each computing device includes multiple computing nodes, the second data transmission rate of each computing device includes the third data transmission rate corresponding to each of the multiple computing nodes; the third data transmission rate is the data transmission rate between each computing node and the memory; the video memory usage of each computing device includes the video memory usage corresponding to each of the multiple computing nodes; the computing resource utilization rate of each computing device includes the computing resource utilization rate corresponding to each of the multiple computing nodes; accordingly, please refer to Figure 6 , which shows a method for processing status data of a computing device including multiple computing nodes, the method may include:

[0096] S610. Sum the video memory usage corresponding to each of the multiple computing nodes of each computing device at the current time node to obtain a first summation result.

[0097] For each computing device, the memory usage of each computing node is recorded as V, so the calculation formula of the first summation result is shown in formula (3):

[0098] The first summation result = ∑V j (3)

[0099] Wherein, j is the count of computing power nodes, and 0≤j≤total number of computing power nodes-1, where the total number of computing power nodes refers to the total number of computing power nodes in the current computing power device.

[0100] S620. Sum the computing resource usage rates corresponding to the multiple computing nodes of each computing device at the current time node to obtain a second summation result.

[0101] For each computing device, the computing resource utilization rate of each computing node is recorded as R, so the calculation formula of the second summation result is shown in formula (4):

[0102] Second summation result = ∑R j (4)

[0103] Wherein, j is the count of computing power nodes, and 0≤j≤total number of computing power nodes-1, where the total number of computing power nodes refers to the total number of computing power nodes in the current computing power device.

[0104] S630. Perform a product operation based on the first summation result and the second summation result to obtain a product operation result of each computing power device.

[0105] By summing up the video memory usage and computing resource utilization of multiple computing nodes, the product of the video memory usage and computing resource utilization of each computing device can be determined.

[0106] S640. Perform a sum operation on the third data transmission rates corresponding to each of the multiple computing nodes of each computing device at the current time node to obtain a third sum result.

[0107] For each computing device, the third data transmission rate of each computing node is recorded as S, so the calculation formula of the third summation result is shown in formula (4):

[0108] The third summation result = ∑S j (4)

[0109] Wherein, j is the count of computing power nodes, and 0≤j≤total number of computing power nodes-1, where the total number of computing power nodes refers to the total number of computing power nodes in the current computing power device.

[0110] S650. Based on the first data transmission rate of each computing power device at the current time node and the third summation result, the data transmission rate of each computing power device at the current time node is obtained.

[0111] In this embodiment, each computing device may include multiple computing nodes, and the number of computing nodes included in different computing devices may be the same or different. In the case where each computing device includes multiple computing nodes, the overall operating status data of the corresponding computing device may be determined based on the sum of the operating status data of each computing node, thereby improving the accuracy and comprehensiveness of the determination of the operating status data of the computing device.

[0112] In addition, according to the above content, it can be known that the first data transmission rate refers to the data transmission rate between each computing power device and other computing power devices. For example, it may specifically refer to the network throughput of each computing power device. That is, the first data transmission rate itself represents the network throughput of a single computing power device, and there is no need to sum it up based on the data corresponding to each computing power node.

[0113] In one example, a state fluctuation analysis is performed on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result, including:

[0114] Determine the data volume fluctuation range based on the target data volume processed by each computing device at the current time node and the target data volume processed by each computing device at the historical time node;

[0115] The second fluctuation analysis result is determined based on the data volume fluctuation amplitude.

[0116] That is, in this embodiment, when performing state fluctuation analysis on each computing power device, data volume fluctuation analysis can be performed based on the target processing data volume of each computing power device at the current time node and the target processing data volume of each computing power device at the historical time node, thereby determining the corresponding second fluctuation analysis result. The historical time node can be the previous time node of the current time node, or the historical time node can be multiple historical time nodes located before the current time node.

[0117] When the historical time node is the previous time node, the eighth difference between the target processing data volume of each computing power device at the current time node and the target processing data volume at the previous time node can be calculated, and the data volume fluctuation range can be determined based on the absolute value of the eighth difference.

[0118] When the historical time nodes are multiple historical time nodes located before the current time node, the sum of the target processing data volumes of the multiple historical time nodes can be calculated, and based on the sum of the target processing data volumes of the multiple historical time nodes and the number of computing power devices in the computing power device cluster, the sixth average value is calculated, and then the ninth difference between the target processing data volume of the current time node and the sixth average value is calculated, and the data volume fluctuation amplitude is determined based on the absolute value of the ninth difference.

[0119] The data volume fluctuation amplitude in this embodiment can be directly determined by the absolute value of the difference as the data volume fluctuation amplitude, or by the ratio of the absolute value of the difference to the target processing data volume at the historical time node as the data volume fluctuation amplitude, or by the ratio of the absolute value of the difference to the average value of the target processing data volume at the historical time node as the data volume fluctuation amplitude.

[0120] Correspondingly, the second fluctuation analysis result may include the above-mentioned data volume fluctuation amplitude. The target processing data volume of each computing power device at the current time node may be known through the second fluctuation analysis result, thereby facilitating the determination of abnormal computing power devices based on the second fluctuation analysis result.

[0121] By performing fluctuation analysis on the target processing data volume of each computing power device at the current time node and the target processing data volume at the historical time node, the data volume fluctuation range of each computing power device is determined. Since the target processing data volume can represent the amount of data that each computing power device can process per unit of video memory and per unit of computing power, if the data volume fluctuation range is abnormal, it can indicate that the corresponding computing power device has an abnormality, thereby improving the convenience and efficiency of subsequent abnormal device positioning.

[0122] Furthermore, in this embodiment, when the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data at the historical time node is greater than the first preset amplitude, the target processing data volume of each computing power device at the historical time node is determined based on the transmission status data of each computing power device at the historical time node and the computing power status data of each computing power device at the historical time node.

[0123] Specifically, the historical time node is the previous time node. If the first fluctuation analysis result of multiple computing power devices in the computing power device cluster at the previous time node indicates that the fluctuation amplitude of the transmission status data of the previous time node is less than or equal to the first preset amplitude, there is no need to perform subsequent operations to locate the abnormal computing power device, and thus there is no need to execute the steps: based on the transmission status data of each computing power device at the previous time node and the computing power status data of each computing power device at the previous historical time node, determine the target processing data volume of each computing power device at the previous time node. If the first fluctuation analysis results of multiple computing devices in the computing power device cluster at the current time node indicate that the fluctuation amplitude of the transmission status data at the current time node is greater than the first preset amplitude, it is necessary to perform subsequent abnormal computing power device positioning operations. The target processing data volume of each computing power device at the current time node and the target processing data volume of each computing power device at the previous time node are required during the abnormal computing power device positioning operation. At this time, the following steps can be executed: Based on the transmission status data of each computing power device at the previous time node and the computing power status data of each computing power device at the previous historical time node, determine the target processing data volume of each computing power device at the previous time node.

[0124] That is, when the transmission status data fluctuation of the computing power device cluster at the previous time node is within the normal fluctuation range, and the transmission status data fluctuation at the current time node is within the abnormal fluctuation range, the target processing data volume of each computing power device at the previous time node and the target processing data volume of each computing power device at the current time node can be calculated at the same time at the current time node, that is, the corresponding calculation is performed when needed, avoiding the situation where the target processing data volume of each computing power device is calculated at each time node but not needed at the time node, thereby saving computing resources.

[0125] It should be noted that any method in this embodiment can be combined based on actual implementation conditions and have corresponding beneficial effects, which will not be described in detail here.

[0126] See also Figure 7, which shows a schematic diagram of the data processing architecture of the present application, which can correspond to the architecture of the cluster management end, wherein the data processing architecture includes a data collection layer and a data processing layer, and the data processing layer includes a storage module and an alarm module; the real-time data collection layer mainly completes the collection of data in four dimensions, the first data transmission rate, the second data transmission rate, the usage of video memory and the utilization rate of computing resources, which can specifically correspond to the network throughput of the computing device, the data transmission rate between the memory and the video memory, the usage of the video memory of the GPU device and the utilization rate of the GPU. The GPU utilization rate and the usage of the video memory can be obtained through the Nvidia cuda library interface. Since the obtained utilization rate and video memory size are single-card dimensions, the present application can be based on the granularity of a single machine. For the GPU utilization rate and the usage of the video memory, the summation method can be used to measure the GPU load of the entire computing device, where j ranges from 0-7, because the number of GPU cards in a single machine is usually 8 cards; R represents the utilization rate of each card, and V represents the usage of the video memory of each card. The data transfer rate between the memory and the video memory can be obtained through the nvidia driver interface, and the data is also based on the card granularity, and needs to be summed up. The value of j is also 0-7, and S indicates the data transfer rate between the video memory and the memory of each card. Since network communication usually goes in and out of the same network, the collected network throughput basically belongs to the network throughput of the entire computing device. The data collected in the four dimensions will be stored in the data storage table. The data storage table is designed as follows:

[0127] Table 1

[0128]

[0129] This application can collect data based on minute-level, distinguish devices by IP, and store the utilization rate, video memory usage and bus bandwidth of each card in the data table. The data of the whole machine IP granularity will be summarized and stored in the data table. The stored data will be sent to the alarm module to calculate the throughput efficiency, and complete the normal operation of the computing power and the alarm output of abnormal throughput nodes based on the calculated value.

[0130] The alarm module is responsible for summarizing the collected data every minute. The summarized data items are the minute-level machine load data: GPU utilization, video memory usage, computing device network throughput, and data transfer rate between memory and video memory. First, the computing device network throughput and data transfer rate between memory and video memory will be summarized and summed according to the granularity of the whole machine. In addition, the alarm module will make a fluctuation trend chart of the operating status of the computing device cluster every minute. Please refer to Figure 8, each point is the summary of the data throughput of the computing power equipment cluster, and the horizontal axis is the time axis; the alarm module will calculate the fluctuation of each point and the adjacent points. If there is a% fluctuation between the point and the adjacent points, it means that there is an abnormal fluctuation in the operation state of the point, and an alarm output is required. The resource operation and maintenance personnel will follow up to locate the problem of the abnormal computing power equipment. In order to further locate the abnormal computing power equipment, the target processing data volume of each computing power equipment can be calculated, that is, the data throughput of the computing power per G memory unit of each computing power equipment, to generate the following Fig. 9 As shown in the target processing data volume fluctuation graph, if the fluctuation rate between the current node and the adjacent node exceeds a%, the alarm is output to the resource operation and maintenance personnel, and the abnormal computing power equipment can be directly located. After finding the abnormal computing power equipment, the dimension data of each card in the data table is compared horizontally. The horizontal comparison method adopts the mean of each dimension, and then the mean is compared with the dimension value of each card. If it is lower than the average value, an exception is thrown. In this way, the abnormal card can be found and the abnormal dimension can be located. For example, for the GPU utilization dimension, the avg (GPU0-GPU7) value is obtained, and then the utilization of each GPU card is compared with the avg value. If it is lower than the avg value, it is recorded as an abnormality of the card and the corresponding dimension. This can assist the resource operation and maintenance personnel to quickly locate the abnormal card point.

[0131] For large model task training scenarios, please refer to Fig.10 , which shows a flow chart of the operation status observation method of the computing power equipment cluster, in which the throughput calculation of the large model per minute will be summarized and calculated by the alarm module, and it will be judged with the adjacent time points whether there will be a% fluctuation. If not, it is recorded as the normal operation of the task in the current minute, and no intervention is required; if the fluctuation range exceeds a%, each GPU node in the minute calculates the throughput of the unit computing power under a single G video memory, and compares the fluctuation range of each machine. If the fluctuation range of each machine is less than a%, but the throughput of the large model fluctuates by a%, an alarm is required to manually intervene and analyze the cause; if the fluctuation of some machines exceeds a%, this solution will compare the distribution of the mean values ​​of the four dimensional data of each card of the single machine based on the data storage table. If there is no situation below the mean, manual intervention is required to analyze the cause. If a dimension value below the mean is hit, the GPU node-card number-dimensional information is alarmed to accurately locate the abnormal point, that is, to assist resource operation and maintenance personnel in locating the operation state failure problem.

[0132] This application can be used for scenarios where large models are trained. Due to the large number of parameters in large model training and the high scale of data exchange during training, the required GPU computing power will reach more than 10,000 cards. When running training tasks on such a large-scale GPU computing power device, how to measure whether the GPU computing power is running normally, and how to assist computing power operation and maintenance personnel in locating abnormal communication points under abnormal circumstances, are urgent problems to be solved in large-model AI training. This application starts from the single-machine throughput of collected training data, and combines the data transmission rate between memory and video memory, the usage data of video memory, and the utilization rate of GPU cards to construct an AI distributed throughput efficiency model, and extracts the throughput of computing power processing per single G video memory unit. It can quantify the computing power processing performance of the running state in real time, which is helpful for the work of resource operation and maintenance personnel and converges the fault problems at the resource layer; it also helps to improve the operational efficiency of resources, facilitate the discovery, analysis and location of problems, and eliminate the time-consuming communication between application researchers and resource operation and maintenance personnel to locate problems.

[0133] See also Fig.11 This embodiment further provides a data processing device, including:

[0134] The operation status data acquisition module 1110 is used to obtain the operation status data of multiple computing devices in the computing device cluster at the current time node; the operation status data includes transmission status data representing the data transmission status of each computing device, and computing power status data representing the computing power usage status of each computing device;

[0135] A first fluctuation analysis module 1120, configured to perform state fluctuation analysis on the multiple computing devices based on the transmission state data of the multiple computing devices at the current time node to obtain a first fluctuation analysis result;

[0136] A first determination module 1130 is used to determine the target processing data volume of each computing power device at the current time node based on the transmission status data of each computing power device at the current time node and the computing power status data of each computing power device at the current time node, when the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data relative to the historical time node is greater than a first preset amplitude; the target processing data volume represents the data processing capability of the corresponding computing power device;

[0137] A second fluctuation analysis module 1140 is used to perform a state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result;

[0138] The second determination module 1150 is used to determine an abnormal computing power device from the multiple computing power devices based on the second fluctuation analysis result; the fluctuation amplitude of the target processing data volume of the abnormal computing power device at the current time node relative to the target processing data volume of the abnormal computing power device at the historical time node is greater than a second preset amplitude.

[0139] Furthermore, each computing device includes a plurality of computing nodes, the transmission status data of each computing device includes the transmission status data corresponding to each of the plurality of computing nodes, and the computing status data of each computing device includes the computing status data corresponding to each of the plurality of computing nodes; the device further includes:

[0140] A first average processing module, configured to perform an average processing based on the transmission status data corresponding to each of the plurality of computing power nodes to obtain average transmission status data;

[0141] A second average processing module is used to perform an average processing based on the computing power status data corresponding to each of the plurality of computing power nodes to obtain average computing power status data;

[0142] A first comparison module, used to compare the transmission status data of each computing power node with the average transmission status data to obtain a first comparison result;

[0143] A second comparison module is used to compare the computing power status data of each computing power node with the average computing power status data to obtain a second comparison result;

[0144] An abnormal computing power node determination module is used to determine an abnormal computing power node from multiple computing power nodes of the abnormal device based on the first comparison result and the second comparison result; the first comparison result of the abnormal computing power node indicates that the transmission status data of the abnormal computing power node is less than the average transmission status data, and / or the second comparison result of the abnormal computing power node indicates that the computing power status data of the abnormal computing power node is less than the average computing power status data.

[0145] Further, the transmission status data of each computing power device includes a first data transmission rate between each computing power device and other computing power devices, and a second data transmission rate between a video memory and a memory in each computing power device, wherein the other computing power devices are computing power devices other than the current computing power device in the multiple computing power devices; the first fluctuation analysis module includes:

[0146] A first summing module is used to perform summing processing based on a first data transmission rate of each computing power device at the current time node and a second data transmission rate of each computing power device at the current time node to obtain a data transmission rate of each computing power device at the current time node;

[0147] A total data transmission rate determination module, used to determine the sum of the data transmission rates of the multiple computing devices at the current time node as the total data transmission rate of the current time node;

[0148] A transmission rate fluctuation amplitude determination module, used to determine the transmission rate fluctuation amplitude based on the total data transmission rate at the current time node and the total data transmission rate at the historical time node;

[0149] The third determination module is used to determine the first fluctuation analysis result based on the transmission rate fluctuation amplitude.

[0150] Furthermore, the computing power status data of each computing power device includes the usage of video memory of each computing power device and the computing power resource utilization rate of each computing power device; the first determination module includes:

[0151] A first product operation module is used to perform a product operation based on the video memory usage of each computing device at the current time node and the computing resource utilization rate of each computing device at the current time node to obtain a product operation result of each computing device;

[0152] The target processing data volume determination module is used to determine the target processing data volume of each computing power device at the current time node based on the data transmission rate of each computing power device at the current time node and the ratio of the multiplication operation result; the target processing data volume represents the amount of data that can be processed by unit video memory and unit computing power.

[0153] Further, each computing device includes a plurality of computing nodes, and the second data transmission rate of each computing device includes the third data transmission rate corresponding to each of the plurality of computing nodes; the third data transmission rate is the data transmission rate between each computing node and the memory; the video memory usage of each computing device includes the video memory usage corresponding to each of the plurality of computing nodes; the computing resource usage rate of each computing device includes the computing resource usage rate corresponding to each of the plurality of computing nodes;

[0154] The first product operation module includes:

[0155] A second summing module is used to sum the video memory usage corresponding to each of the multiple computing nodes of each computing device at the current time node to obtain a first summing result;

[0156] A third summing module is used to sum the computing resource usage rates corresponding to the multiple computing nodes of each computing device at the current time node to obtain a second summing result;

[0157] A second product operation module, used for performing a product operation based on the first summation result and the second summation result to obtain a product operation result of each computing power device;

[0158] The first summation module includes:

[0159] A fourth summing module, configured to sum the third data transmission rates respectively corresponding to the multiple computing nodes of each computing device at the current time node to obtain a third summing result;

[0160] The fifth summation module is used to obtain the data transmission rate of each computing power device at the current time node based on the first data transmission rate of each computing power device at the current time node and the third summation result.

[0161] Furthermore, the second fluctuation analysis module includes:

[0162] A data volume fluctuation range determination module, used to determine the data volume fluctuation range based on the target processing data volume of each computing power device at the current time node and the target processing data volume of each computing power device at the historical time node;

[0163] The fourth determination module is used to determine the second fluctuation analysis result based on the data volume fluctuation amplitude.

[0164] Furthermore, the device also includes:

[0165] The fifth determination module is used to determine the target processing data volume of each computing power device at the historical time node based on the transmission status data of each computing power device at the historical time node and the computing power status data of each computing power device at the historical time node, when the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data relative to the historical time node is greater than a first preset amplitude.

[0166] The device provided in the above embodiment can execute the method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the above embodiment, please refer to the method provided in any embodiment of the present application.

[0167] This embodiment further provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. The at least one instruction or at least one program is loaded by a processor and executed as any of the above methods of this embodiment.

[0168] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above methods.

[0169] Fig.12 is a block diagram of an electronic device for data processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Fig.12 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0170] Those skilled in the art will understand that Fig.12 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on conventional or non-creative labor. The steps and sequence listed in the embodiments are only one way of executing the sequence of many steps and do not represent the only execution sequence. When the actual system or interrupt product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment).

[0172] The structure shown in this embodiment is only a partial structure related to the scheme of the present application, and does not constitute a limitation on the device to which the scheme of the present application is applied. The specific device may include more or fewer components than shown, or combine certain components, or have different arrangements of components. It should be understood that the method, device, etc. disclosed in this embodiment can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the modules is only a division of a logical function. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or unit modules.

[0173] Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.

[0174] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Obtain the operating status data of multiple computing devices in the computing device cluster at the current time node; The operation status data includes transmission status data representing the data transmission status of each computing power device, and computing power status data representing the computing power usage status of each computing power device; Performing a state fluctuation analysis on the computing power device cluster based on the transmission state data of the multiple computing power devices at the current time node to obtain a first fluctuation analysis result; In a case where the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data of the historical time node is greater than the first preset amplitude, based on the transmission status data of each computing power device at the current time node and the computing power status data of each computing power device at the current time node, determining the target processing data volume of each computing power device at the current time node; The target processing data volume represents the data processing capability of the corresponding computing device; Performing a state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result; Based on the second fluctuation analysis result, determining an abnormal computing power device from the multiple computing power devices; The fluctuation range of the target processing data volume of the abnormal computing power device at the current time node relative to the target processing data volume of the abnormal computing power device at the historical time node is greater than the second preset range.

2. The method according to claim 1, characterized in that Each computing device includes a plurality of computing nodes, the transmission status data of each computing device includes the transmission status data corresponding to each of the plurality of computing nodes, and the computing status data of each computing device includes the computing status data corresponding to each of the plurality of computing nodes; After determining an abnormal computing power device from the plurality of computing power devices based on the second fluctuation analysis result, the method further includes: Performing an averaging process based on the transmission status data corresponding to each of the plurality of computing power nodes to obtain average transmission status data; Performing an averaging process based on the computing power status data corresponding to each of the multiple computing power nodes to obtain average computing power status data; Compare the transmission status data of each computing power node with the average transmission status data to obtain a first comparison result; Compare the computing power status data of each computing power node with the average computing power status data to obtain a second comparison result; Based on the first comparison result and the second comparison result, an abnormal computing power node is determined from the multiple computing power nodes of the abnormal device; the first comparison result of the abnormal computing power node indicates that the transmission status data of the abnormal computing power node is smaller than the average transmission status data, and / or the second comparison result of the abnormal computing power node indicates that the computing power status data of the abnormal computing power node is smaller than the average computing power status data.

3. The method according to claim 1, characterized in that The transmission status data of each computing device includes a first data transmission rate between each computing device and other computing devices, and a second data transmission rate between a video memory and a memory in each computing device, wherein the other computing devices are computing devices other than the current computing device among the multiple computing devices; The performing state fluctuation analysis on the multiple computing power devices based on the transmission state data of the multiple computing power devices at the current time node to obtain a first fluctuation analysis result includes: Based on the sum of the first data transmission rate of each computing device at the current time node and the second data transmission rate of each computing device at the current time node, the data transmission rate of each computing device at the current time node is obtained; Determine the sum of the data transmission rates of the multiple computing devices at the current time node as the total data transmission rate of the current time node; Determine the transmission rate fluctuation range based on the total data transmission rate at the current time node and the total data transmission rate at the historical time nodes; The first fluctuation analysis result is determined based on the transmission rate fluctuation amplitude.

4. The method according to claim 3, characterized in that The computing power status data of each computing power device includes the video memory usage of each computing power device and the computing power resource utilization rate of each computing power device; The determining, based on the transmission status data of each computing device at the current time node and the computing status data of each computing device at the current time node, a target processing data volume of each computing device at the current time node, includes: Perform a product operation based on the video memory usage of each computing device at the current time node and the computing resource utilization rate of each computing device at the current time node to obtain a product operation result of each computing device; Based on the data transmission rate of each computing power device at the current time node and the ratio of the multiplication operation result, the target processing data volume of each computing power device at the current time node is determined; the target processing data volume represents the amount of data that can be processed by unit video memory and unit computing power.

5. The method according to claim 4, characterized in that Each computing device includes multiple computing nodes, and the second data transmission rate of each computing device includes the third data transmission rate corresponding to each of the multiple computing nodes; the third data transmission rate is the data transmission rate between each computing node and the memory; the video memory usage of each computing device includes the video memory usage corresponding to each of the multiple computing nodes; the computing resource utilization rate of each computing device includes the computing resource utilization rate corresponding to each of the multiple computing nodes; The product operation is performed based on the memory usage of each computing device at the current time node and the computing resource utilization rate of each computing device at the current time node to obtain the product operation result of each computing device, including: Performing a sum operation on the video memory usage corresponding to each of the multiple computing nodes of each computing device at the current time node to obtain a first sum result; Summing the computing resource usage rates corresponding to the multiple computing nodes of each computing device at the current time node to obtain a second summation result; Perform a product operation based on the first summation result and the second summation result to obtain a product operation result of each computing power device; The data transmission rate of each computing device at the current time node is obtained by summing the first data transmission rate of each computing device at the current time node and the second data transmission rate of each computing device at the current time node, including: Performing a sum operation on the third data transmission rates respectively corresponding to the multiple computing nodes of each computing device at the current time node to obtain a third sum result; Based on the first data transmission rate of each computing power device at the current time node and the third summation result, the data transmission rate of each computing power device at the current time node is obtained.

6. The method according to claim 1 or 4, characterized in that: The performing state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result includes: Determine the data volume fluctuation range based on the target data volume processed by each computing device at the current time node and the target data volume processed by each computing device at the historical time node; The second fluctuation analysis result is determined based on the data volume fluctuation amplitude.

7. The method according to claim 1, characterized in that The method further comprises: When the first fluctuation analysis result represents the transmission status data of the current time node, and the fluctuation amplitude of the transmission status data of the historical time node is greater than the first preset amplitude, the target processing data volume of each computing power device at the historical time node is determined based on the transmission status data of each computing power device at the historical time node and the computing power status data of each computing power device at the historical time node.

8. A data processing device, characterized in that: include: The operation status data acquisition module is used to obtain the operation status data of multiple computing devices in the computing device cluster at the current time node; The operation status data includes transmission status data representing the data transmission status of each computing power device, and computing power status data representing the computing power usage status of each computing power device; A first fluctuation analysis module, configured to perform state fluctuation analysis on the multiple computing power devices based on the transmission state data of the multiple computing power devices at the current time node to obtain a first fluctuation analysis result; A first determination module is used to determine the target processing data volume of each computing power device at the current time node based on the transmission status data of each computing power device at the current time node and the computing power status data of each computing power device at the current time node, when the first fluctuation analysis result represents that the fluctuation amplitude of the transmission status data of the current time node relative to the transmission status data of the historical time node is greater than a first preset amplitude; The target processing data volume represents the data processing capability of the corresponding computing device; A second fluctuation analysis module is used to perform a state fluctuation analysis on each computing power device based on the target processing data volume of each computing power device at the current time node to obtain a second fluctuation analysis result; A second determination module, configured to determine an abnormal computing power device from the plurality of computing power devices based on the second fluctuation analysis result; The fluctuation range of the target processing data volume of the abnormal computing power device at the current time node relative to the target processing data volume of the abnormal computing power device at the historical time node is greater than the second preset range.

9. An electronic device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by a processor and executes the data processing method according to any one of claims 1 to 7.

Images