Operation anomaly detection method and device for intelligent computing power network

Abstract

The invention relates to an operation anomaly detection method and device for an intelligent computing power network. The method comprises the following steps: splitting original operation data of a target job in an intelligent computing power network in a to-be-detected time period to obtain local operation data corresponding to each sub-time period in the to-be-detected time period, determining an abnormal sub-time period from each sub-time period according to the similarity between the local operation data corresponding to two adjacent sub-time periods, and sending the abnormal sub-time period to the intelligent computing power network; and obtaining historical operation data in a historical detection time period corresponding to the abnormal sub-time period, and determining an abnormal detection result of the target job according to the historical operation data and the local operation data corresponding to the abnormal sub-time period. By adopting the method, the accuracy of operation anomaly detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for detecting anomalies in an intelligent computing network. Background Art

[0002] With the expansion of computing power scale and the widespread application of intelligent computing power networks, in the process of job processing in intelligent computing power networks, in order to avoid resource waste due to job failure, it is necessary to perform anomaly detection on the operation of jobs.

[0003] The existing job anomaly detection methods generally directly apply the anomaly detection methods for jobs in ordinary networks to jobs in intelligent computing networks. For example, a preset threshold is used to monitor a certain type of operation data.

[0004] However, since the types of jobs in ordinary networks are relatively single, directly applying the anomaly detection method for jobs in ordinary networks to jobs in intelligent computing networks is prone to misjudgment, thereby reducing the accuracy of job anomaly detection. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device for job anomaly detection in an intelligent computing network that can improve the accuracy of job anomaly detection in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for detecting anomalies in a job of an intelligent computing network, comprising:

[0007] The original operation data of the target job in the intelligent computing network during the period to be detected is split to obtain the local operation data corresponding to each sub-period within the period to be detected;

[0008] Determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods;

[0009] Obtain historical operation data within the historical detection period corresponding to the abnormal sub-period;

[0010] Determine the anomaly detection result of the target job based on the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0011] In the embodiments of the present application, on the one hand, by determining the abnormal sub-period from each sub-period based on the similarity between the local operating data corresponding to two adjacent sub-periods, the rationality of the determination of the abnormal sub-period can be ensured; on the other hand, by performing further abnormality verification on the local operating data corresponding to the abnormal sub-period based on historical operating data, it is possible to avoid misjudging reasonable data fluctuations as operation abnormalities, thereby improving the accuracy of the abnormality detection results.

[0012] In one embodiment, the original operation data of the target job in the intelligent computing network during the period to be detected is split to obtain the local operation data corresponding to each sub-period within the period to be detected, including:

[0013] Perform first-order difference processing on the original operation data of the target job in the intelligent computing network during the period to be detected to obtain the global operation data of the target job during the period to be detected; split the global operation data to obtain the local operation data corresponding to each sub-period within the period to be detected.

[0014] In an embodiment of the present application, by performing first-order difference processing on the original operation data of the target job, the global operation data after removing the change trend is obtained, and then the global operation data is split to obtain the local operation data of each part, which can make the local operation data more stable, thereby improving the accuracy of anomaly detection.

[0015] In one embodiment, determining an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods includes:

[0016] For each sub-period, the similarity score of the sub-period is determined according to the similarity between the local operating data corresponding to the sub-period and the local operating data corresponding to the neighbor sub-period; wherein the neighbor sub-period is a sub-period adjacent to the sub-period in each sub-period, and the neighbor sub-period is located before or after the sub-period; according to the similarity score of each sub-period, the abnormal sub-period is determined from each sub-period.

[0017] In the embodiment of the present application, by determining a similarity score based on the similarity between adjacent local operation data, and determining an abnormal sub-period from each sub-period based on the similarity score of each sub-period, the rationality of determining the abnormal sub-period can be ensured.

[0018] In one embodiment, determining a similarity score of a sub-period according to a similarity between local operation data corresponding to the sub-period and local operation data corresponding to a neighboring sub-period includes:

[0019] Determine the data change trend of the local operation data corresponding to the sub-period and the data change trend of the local operation data corresponding to the neighboring sub-period; determine the similarity score of the sub-period according to the trend similarity between the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period.

[0020] In the embodiment of the present application, by determining the similarity score of the sub-period according to the trend similarity between the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period, the accuracy of the similarity score determination can be ensured.

[0021] In one embodiment, determining an abnormal sub-period from each sub-period according to the similarity score of each sub-period includes:

[0022] From the similarity scores of each sub-period, an outlier similarity score is selected; and the sub-period corresponding to the outlier similarity score is taken as an abnormal sub-period.

[0023] In the embodiment of the present application, by selecting the outlier similarity score that deviates from the overall similarity score of each sub-period, and directly using the sub-period corresponding to the outlier similarity score as the abnormal sub-period, the rationality of the determination of the abnormal sub-period can be ensured.

[0024] In one embodiment, obtaining historical operation data in a historical detection period corresponding to an abnormal sub-period includes:

[0025] According to the time characteristics corresponding to the abnormal sub-period, the historical detection period corresponding to the abnormal sub-period is determined; from each candidate operation data within the historical detection period, the historical operation data that meets the operation characteristics of the target operation and is in a normal operation state is selected.

[0026] In an embodiment of the present application, by determining the historical detection period based on the time feature dimension, and screening out the historical operation data in the normal operation state based on the operation characteristics of the target operation, the rationality and accuracy of the determination of the historical operation data can be ensured.

[0027] In one embodiment, determining an anomaly detection result of a target job according to the historical operation data and the local operation data corresponding to the abnormal sub-period includes:

[0028] Determine the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period; and determine the abnormality detection result of the target job based on the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0029] In the embodiment of the present application, by determining the abnormality detection result of the target job based on the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period, the accuracy of the abnormality detection result can be guaranteed.

[0030] In a second aspect, the present application also provides a job anomaly detection device for an intelligent computing network, comprising:

[0031] The data splitting module is used to split the original operation data of the target job in the intelligent computing network within the period to be detected, and obtain the local operation data corresponding to each sub-period within the period to be detected;

[0032] an abnormality determination module, used to determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods;

[0033] A data acquisition module is used to acquire historical operation data in a historical detection period corresponding to an abnormal sub-period;

[0034] The anomaly detection module is used to determine the anomaly detection result of the target job based on the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0035] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] The original operation data of the target job in the intelligent computing network during the period to be detected is split to obtain the local operation data corresponding to each sub-period within the period to be detected;

[0037] Determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods;

[0038] Obtain historical operation data within the historical detection period corresponding to the abnormal sub-period;

[0039] Determine the anomaly detection result of the target job based on the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] The original operation data of the target job in the intelligent computing network during the period to be detected is split to obtain the local operation data corresponding to each sub-period within the period to be detected;

[0042] Determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods;

[0043] Obtain historical operation data within the historical detection period corresponding to the abnormal sub-period;

[0044] Determine the anomaly detection result of the target job based on the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0045] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0046] The original operation data of the target job in the intelligent computing network during the period to be detected is split to obtain the local operation data corresponding to each sub-period within the period to be detected;

[0047] Determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods;

[0048] Obtain historical operation data within the historical detection period corresponding to the abnormal sub-period;

[0049] Determine the anomaly detection result of the target job based on the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0050] The above-mentioned method and device for detecting abnormal operation of a job in an intelligent computing network splits the original operation data of the target job in the intelligent computing network in the period to be detected, obtains the local operation data corresponding to each sub-period in the period to be detected, and determines the abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods; then, determines the abnormal detection result of the target job according to the historical operation data in the historical detection period corresponding to the abnormal sub-period and the local operation data corresponding to the abnormal sub-period. Compared with the related art, which only uses a preset data threshold to monitor the operation status of the job, the above-mentioned method, on the one hand, can ensure the rationality of the determination of the abnormal sub-period by determining the abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods; on the other hand, by further performing abnormality verification on the local operation data corresponding to the abnormal sub-period based on the historical operation data, it can avoid misjudging reasonable data fluctuations as job abnormalities, thereby improving the accuracy of the abnormal detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a method for detecting abnormality of a job in an intelligent computing network in one embodiment;

[0053] Figure 2 A schematic diagram of abnormal operation detection in one embodiment;

[0054] Figure 3 A schematic diagram of a process for determining an abnormal sub-period in one embodiment;

[0055] Figure 4 A schematic diagram of a process for determining a similarity score in one embodiment;

[0056] Figure 5 A schematic diagram of a process for determining a similarity score in another embodiment;

[0057] Figure 6 A schematic diagram of a process for obtaining historical operation data in one embodiment;

[0058] Figure 7 A schematic diagram of a process for determining an abnormality detection result in one embodiment;

[0059] Figure 8 It is a flowchart of a method for detecting abnormality of a job in an intelligent computing network in another embodiment;

[0060] Fig. 9 is a structural block diagram of a job anomaly detection device for an intelligent computing power network in one embodiment;

[0061] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] With the expansion of computing power scale and the widespread application of intelligent computing power networks, in the process of job processing in intelligent computing power networks, in order to avoid resource waste due to job failure, it is necessary to perform anomaly detection on the operation of jobs.

[0064] The existing job anomaly detection methods generally directly apply the anomaly detection methods for jobs in ordinary networks to jobs in intelligent computing networks. For example, a preset threshold is used to monitor a certain type of operation data.

[0065] However, since the types of jobs in ordinary networks are relatively single, directly applying the anomaly detection method for jobs in ordinary networks to jobs in intelligent computing networks is prone to misjudgment, thereby reducing the accuracy of job anomaly detection.

[0066] Based on this, in an exemplary embodiment, Figure 1 As shown, a method for detecting anomalies in a job of an intelligent computing network is provided. The method is described by taking the application of the method to an anomaly detection device deployed in the intelligent computing network as an example. The method specifically includes the following steps:

[0067] S101, splitting the original operation data of the target job in the intelligent computing network within the period to be detected to obtain the local operation data corresponding to each sub-period within the period to be detected.

[0068] Among them, the target job is a job with anomaly detection requirements in the intelligent computing network; the period to be detected is the period when anomaly detection is required; the original operation data is the data generated when the target job runs during the period to be detected; the so-called local operation data is the operation data in each sub-period; the duration of each sub-period is the same.

[0069] In an embodiment of the present application, a user can submit each job to the intelligent computing power network through a user interface displayed by a terminal to run each job in the intelligent computing power network.

[0070] At the same time, in order to more accurately monitor the running status of each job running in the intelligent computing network, in an optional implementation, the user can configure an anomaly detection strategy for the job through the administrator interface displayed by the terminal.

[0071] Exemplary, reference Figure 2 As shown in the diagram of job anomaly detection, users can modify the modifiable options in the monitoring strategy displayed on the administrator page to configure an anomaly detection strategy for the job. Among them, the modifiable options include but are not limited to the time window size (i.e. the time window for anomaly detection), the number of collections (i.e. the number of data collections within the time window), the anomaly detection interval (i.e. the interval to be detected can be customized), the data downsampling method (due to the high dimension and large data volume of time series data, a specific data downsampling method can be set to reduce the data volume) and the data source (i.e. the job type of the target job can be customized).

[0072] After configuring the anomaly detection strategy, when it is determined that the time period to be detected is a historical time period, the original operation data of the target job in the time period to be detected can be directly obtained based on the data source and anomaly detection interval in the anomaly detection strategy; then, based on the time window size and the number of collections, the original operation data is split and processed to obtain the local operation data corresponding to each sub-time period in the time period to be detected.

[0073] When it is determined that the time period to be detected is the current time period, the local operation data corresponding to each sub-time period can be collected in real time according to the data source, the time window size and the number of collections.

[0074] In another optional implementation, the user may directly send a job detection request including a job identifier of a target job and a time period to be detected to the anomaly detection device through a request initiation interface displayed by the terminal.

[0075] After detecting the job detection request, the original operation data of the target job in the time period to be detected can be obtained based on the job identifier; then, based on the preset splitting time, the original operation data in the time period to be detected can be split into local operation data corresponding to each sub-time period. The preset splitting time can be determined by the technician based on historical experience or through a large number of experimental results, and this application does not limit this.

[0076] S102: determining an abnormal sub-period from among the sub-periods according to the similarity between the local operation data corresponding to two adjacent sub-periods.

[0077] The abnormal sub-period is used to represent the sub-period in which the target job may run abnormally.

[0078] It is understandable that, since job anomalies usually show long-term trends, sub-intervals with long-term different data distributions can be selected from each sub-interval as abnormal sub-intervals based on the similarity between the local operation data corresponding to each sub-interval.

[0079] In an optional embodiment, the local operating data corresponding to each sub-period can be input into a trained first abnormality determination model in chronological order. The first abnormality determination model determines the abnormal sub-period from each sub-period according to the similarity between the local operating data corresponding to two sub-periods adjacent in time sequence, and outputs it.

[0080] In another optional implementation, since the target job under normal operating conditions should have similar operating characteristics in each sub-period, for each sub-period, it is possible to determine whether the sub-period is an abnormal sub-period by analyzing the local operating data corresponding to the sub-period to see whether there are similar data characteristics between the local operating data corresponding to the neighboring sub-periods that are adjacent in time sequence.

[0081] For example, if there are no similar data features between the local operation data A' corresponding to the sub-period A and the local operation data B' corresponding to the sub-period B, it is proved that there may be abnormal sub-periods in the sub-period A and the sub-period B.

[0082] S103, obtaining historical operation data in a historical detection period corresponding to the abnormal sub-period.

[0083] The so-called historical detection period is used to characterize the period in the historical period that has the same period characteristics as the abnormal sub-period. For example, if the abnormal sub-period is November 15, 2023 to November 30, 2023, the historical detection period can be November 15, 2022 to November 30, 2022. The so-called historical operation data is the actual operation data of the operation in the historical detection period.

[0084] It is understandable that due to the diverse types of jobs executed on the intelligent computing network, each following a unique operating mode, and the fact that these jobs usually run for a long time, it is common for job data to change dramatically (sudden increase or decrease) in a short period of time.

[0085] Based on this, in order to avoid misjudging the above situation as a job abnormality, in an optional embodiment, according to the time period characteristics of the abnormal sub-period, historical operation data in the historical detection period with the same time period characteristics as the abnormal sub-period can be obtained from a database constructed based on the historical operation data of each job to determine whether the abnormal fluctuations in the data within the abnormal sub-period are normal data fluctuations.

[0086] Exemplarily, if the target job is already in a running state during the historical detection period, the actual running data of the target job during the historical detection period is preferentially obtained as the historical running data; if the target job is not in a running state during the historical detection period, the actual running data of other jobs of the same job type as the target job during the historical detection period can be obtained as the historical running data.

[0087] It is understandable that in order to ensure the reliability of data collection, only data under normal operation of the operation may be collected as a forward reference; or only data under abnormal operation of the operation may be collected as a reverse reference.

[0088] S104, determining an abnormality detection result of the target job according to the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0089] The so-called anomaly detection result refers to whether the target job has an anomaly during the detection period. The anomaly detection result can be that the job is running abnormally or that the job is running normally.

[0090] After obtaining the historical running jobs, the historical running data can be used as a reference to determine whether there are abnormalities in the local running data corresponding to the abnormal sub-period, and then determine the abnormality detection result of the target job.

[0091] In an optional implementation, historical operation data, the job operation status corresponding to the historical operation data, and the local operation data corresponding to the abnormal sub-period can be simultaneously input into a trained anomaly judgment model, and the anomaly judgment model outputs the anomaly detection result based on the historical operation data, job operation status, local operation data and model parameters.

[0092] In another optional implementation, the data fluctuations corresponding to the historical operation data and the data fluctuations of the local operation data corresponding to the abnormal sub-period may be compared for consistency. If the comparison result shows a high degree of consistency, it is determined that the operation status of the target job is consistent with the operation status of the job corresponding to the historical operation data; if the comparison result shows a low degree of consistency, it is determined that the operation status of the target job is opposite to the operation status of the job corresponding to the historical operation data.

[0093] For example, when the job operation status of the historical operation data is normal operation of the job, if the data fluctuation of the historical operation data is consistent with the data fluctuation of the local operation data corresponding to the abnormal sub-period, then the anomaly detection result is determined to be normal operation of the job; if the data fluctuation of the historical operation data is inconsistent with the data fluctuation of the local operation data corresponding to the abnormal sub-period, then the anomaly detection result is determined to be abnormal operation of the job.

[0094] If the target job is running abnormally, continue to refer to Figure 2 The operation abnormality detection schematic diagram shown can generate operation abnormality alarm information according to the target operation's detection period and operation identifier, and feed back the abnormality alarm information to the management end to prompt the management personnel to handle the target operation abnormally. At the same time, the local operation data corresponding to the abnormal sub-period can be input into the database as operation abnormality data, which can be used as reference data for the operation in abnormal operation state in the future.

[0095] In the above-mentioned method for detecting job anomalies in an intelligent computing network, the original operation data of the target job in the intelligent computing network in the period to be detected is split to obtain the local operation data corresponding to each sub-period in the period to be detected, and the abnormal sub-period is determined from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods; then, the abnormal detection result of the target job is determined according to the historical operation data in the historical detection period corresponding to the abnormal sub-period and the local operation data corresponding to the abnormal sub-period. Compared with the related art, in which only a preset data threshold is used to monitor the operation status of the job, the above-mentioned method, on the one hand, can ensure the rationality of the determination of the abnormal sub-period by determining the abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods; on the other hand, by further performing anomaly verification on the local operation data corresponding to the abnormal sub-period based on the historical operation data, it is possible to avoid misjudging reasonable data fluctuations as job anomalies, thereby improving the accuracy of the anomaly detection results.

[0096] In order to ensure the accuracy of anomaly detection, based on the above embodiments, in an embodiment of the present application, an optional method of data processing is provided, specifically, first-order difference processing is performed on the original operation data of the target job in the intelligent computing power network during the period to be detected to obtain the global operation data of the target job during the period to be detected; the global operation data is split to obtain the local operation data corresponding to each sub-period within the period to be detected.

[0097] The so-called global operation data is used to represent the overall operation data of the target job during the period to be detected.

[0098] In order to make the operation data more stable, after obtaining the original operation data of the target job in the period to be detected, the original operation data can be directly processed by first-order difference, and the processed original operation data can be used as global operation data to remove the long-term trend in the original operation data.

[0099] In an optional implementation, the global operation data may be split in the time dimension according to a preset time window size and in a chronological order, and the split result may be used as the local operation data corresponding to each sub-period within the time period to be detected. In this case, the sub-periods do not overlap.

[0100] For example, the time period to be detected is 4:00-7:00, and when the time window size is 1 hour, the sub-time periods 4:00-5:00, 5:00-6:00 and 6:00-7:00 can be obtained.

[0101] In another optional implementation, when the duration of the time period to be detected is short, in order to increase the number of sub-periods, the global operation data in the time period to be detected can be split in the time dimension based on the time window size and the window sliding distance to obtain the local operation data corresponding to each sub-period. In this case, the sub-periods overlap.

[0102] For example, the time period to be detected is 4:00-7:00. When the time window size is 1 hour and the window sliding distance is 30 minutes, the sub-time periods 4:00-5:00, 4:30-5:30, 5:00-6:00, 5:30-6:30 and 6:00-7:00 can be obtained.

[0103] In an embodiment of the present application, by performing first-order difference processing on the original operation data of the target job, the global operation data after removing the change trend is obtained, and then the global operation data is split to obtain the local operation data of each part, which can make the local operation data more stable, thereby improving the accuracy of anomaly detection.

[0104] In order to ensure the accuracy of the abnormal sub-period, based on the above embodiment, in the embodiment of the present application, an optional method for determining the abnormal sub-period is provided, such as Figure 3 As shown, the specific steps include:

[0105] S301 , for each sub-period, determining a similarity score of the sub-period according to the similarity between the local operation data corresponding to the sub-period and the local operation data corresponding to the neighboring sub-period.

[0106] The neighbor sub-period is a sub-period adjacent to the sub-period in each sub-period, and the neighbor sub-period is located before or after the sub-period. It is understandable that in order to ensure the rationality of similarity judgment, the neighbor sub-period of each sub-period needs to be defined as a sub-period in the same direction.

[0107] For example, in the time sequence of sub-periods 1, 2, and 3, when the neighbor sub-period is defined before the sub-period, the neighbor sub-period of sub-period 2 is sub-period 1, and the neighbor sub-period of sub-period 3 is sub-period 2; when the neighbor sub-period is defined after the sub-period, the neighbor sub-period of sub-period 1 is sub-period 2, and the neighbor sub-period of sub-period 2 is sub-period 3.

[0108] In the case where there may be no neighboring sub-periods for the first sub-period or the last sub-period, since the local operation data of each sub-period have their data trends removed, the first sub-period and the last sub-period may be regarded as neighboring sub-periods.

[0109] For example, in the time sequence of sub-periods 1, 2, and 3, when the neighbor sub-period is defined before the sub-period, the neighbor sub-period of sub-period 1 is sub-period 3; when the neighbor sub-period is defined after the sub-period, the neighbor sub-period of sub-period 3 is sub-period 1.

[0110] In an optional implementation, for each sub-period, the local operation data corresponding to the sub-period and the local operation data corresponding to the neighboring sub-period of the sub-period can be directly input into a first similarity determination model constructed based on a similarity algorithm, and the first similarity determination model outputs a similarity score for the sub-period based on the local operation data and model parameters in the two sub-periods. The similarity algorithm can be a Fast Dynamic Time Warping (Fast DTW) algorithm.

[0111] In another optional embodiment, for each sub-period, a first data feature of the local operating data corresponding to the sub-period and a second data feature of the local operating data corresponding to the neighboring sub-period of the sub-period can be determined respectively; then, the similarity value between the first data feature and the second data feature is used as the similarity score of the sub-period.

[0112] S302: Determine an abnormal sub-period from each sub-period according to the similarity score of each sub-period.

[0113] In an optional implementation, a standard similarity score may be generated based on the similarity scores of each sub-period. For example, the average of the similarity scores of each sub-period may be used as the standard similarity score.

[0114] Subsequently, for each sub-period, the similarity score of the sub-period can be compared with the standard similarity score. If the difference between the similarity score of the sub-period and the standard similarity score is less than a preset score threshold, the sub-period is determined to be a normal sub-period; if the difference between the similarity score of the sub-period and the standard similarity score is greater than or equal to the preset score threshold, the sub-period is determined to be an abnormal sub-period. The score threshold can be determined based on the historical experience of relevant technical personnel or based on a large number of experimental results, which is not limited in this application.

[0115] In another optional implementation, for each sub-period, whether the sub-period is an abnormal sub-period may be determined based on the similarity score of the sub-period and the deviation between the similarity scores of other sub-periods.

[0116] Exemplarily, if the similarity score of a certain sub-period deviates greatly from the similarity scores of other sub-periods, the sub-period is determined to be an abnormal sub-period; if the similarity score of a certain sub-period deviates slightly from the similarity scores of other sub-periods, the sub-period is determined to be a normal sub-period.

[0117] In the embodiment of the present application, by determining a similarity score based on the similarity between adjacent local operation data, and determining an abnormal sub-period from each sub-period based on the similarity score of each sub-period, the rationality of determining the abnormal sub-period can be ensured.

[0118] In order to ensure the accuracy of the similarity score, based on the above embodiment, in the embodiment of the present application, an optional method for determining the similarity score is provided, such as Figure 4 As shown, the specific steps include:

[0119] S401, determining a data change trend of local operation data corresponding to a sub-period, and a data change trend of local operation data corresponding to a neighboring sub-period.

[0120] The so-called data change trend is used to characterize the change of the local running number at each sampling time.

[0121] In an optional implementation, the local operating data corresponding to the sub-period may be input into a trained trend determination model, and the trend determination model outputs the data change trend corresponding to the sub-period based on the local operating data corresponding to the sub-period and model parameters.

[0122] Correspondingly, the local operation data corresponding to the neighbor sub-period may also be input into a trained trend determination model, and the trend determination model outputs the data change trend corresponding to the neighbor sub-period based on the local operation data corresponding to the neighbor sub-period and model parameters.

[0123] In another optional implementation, the data change trend at each sampling moment in the sub-period can be determined based on the data fluctuation between the running data at each sampling moment in the sub-period; correspondingly, the data change trend at each sampling moment in the neighboring sub-period can also be determined based on the data fluctuation between the running data at each sampling moment in the neighboring sub-period. It is understandable that the number of sampling moments in the sub-period is the same as the number of sampling moments in the neighboring sub-period.

[0124] S402 , determining a similarity score of a sub-period according to a trend similarity between a data change trend corresponding to a sub-period and a data change trend corresponding to a neighboring sub-period.

[0125] In an optional implementation, the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period may be compared for consistency to obtain the trend similarity between the data change trends of the two sub-periods; then, based on the trend similarity, the similarity score of the sub-periods is determined.

[0126] Exemplarily, for each sampling moment in a sub-period, the corresponding sampling moment can be determined from each sampling moment in a neighboring sub-period according to the arrangement result of the sampling moment in the time sequence; then, the trend similarity at the sampling moment is determined according to the data change trend at the sampling moment and the data change trend at the corresponding sampling moment. Further, the trend similarity between the data change trends of two sub-periods can be determined according to the trend similarity at each sampling moment in a sub-period.

[0127] For example, the sub-period T includes sampling moments T1, T2, and T3, and the neighbor sub-period S includes sampling moments S1, S2, and S3. The trend similarity K1 can be determined based on the data change trends under T1 and S1, the trend similarity K2 can be determined based on the data change trends under T2 and S2, and the trend similarity K3 can be determined based on the data change trends under T3 and S3; then, the trend similarity between the sub-period T and the neighbor sub-period S can be determined based on K1, K2, and K3.

[0128] After the trend similarity is determined, the determined trend similarity can be used as an index to query the pre-constructed correspondence between candidate trend similarities and candidate similarity scores, so as to obtain the similarity score of the sub-period.

[0129] In practical applications, there may be a situation where the overall data change trends of local operating data in two sub-periods are consistent, but the data change times are inconsistent. If the above-mentioned method of comparing one by one based on sampling time is adopted, the accuracy of determining the similarity score may be reduced.

[0130] Based on this, in another optional implementation, the FastDTW algorithm can be used to determine the trend similarity between the sub-period and the neighboring sub-period based on the common trend shape between the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period; then, based on the trend similarity, determine the similarity score of the sub-period.

[0131] In the embodiment of the present application, by determining the similarity score of the sub-period according to the trend similarity between the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period, the accuracy of the similarity score determination can be ensured.

[0132] In order to ensure the rationality of the determination of the abnormal sub-period, based on the above embodiment, in the embodiment of the present application, another optional method for determining the similarity score is provided, such as Figure 5 As shown, the specific steps include:

[0133] S501 , selecting an outlier similarity score from the similarity scores of each sub-period.

[0134] The so-called outlier similarity score is used to characterize the similarity score of each sub-period that deviates from other similarity scores.

[0135] In an optional implementation, the similarity score of each sub-period can be mapped to a two-dimensional coordinate system to obtain the similarity score points of each sub-period. Subsequently, each similarity score point is processed for outlier mining to obtain outlier score points that deviate from other similarity score points, and the similarity score corresponding to the outlier score point is used as the outlier similarity score.

[0136] For example, a Z-Score method may be used to determine the degree of deviation between each similarity score point and other similarity score points, thereby determining outlier score points.

[0137] S502: The sub-period corresponding to the outlier similarity score is taken as an abnormal sub-period.

[0138] It can be understood that since each similarity score is associated with a sub-period, and the degree of deviation of the similarity score can characterize the abnormality of the local operating data in the sub-period to a certain extent, therefore, in an optional implementation, after determining the outlier similarity score, the sub-period corresponding to the outlier similarity score can be directly used as the abnormal sub-period.

[0139] In the embodiment of the present application, by selecting the outlier similarity score that deviates from the overall similarity score of each sub-period, and directly using the sub-period corresponding to the outlier similarity score as the abnormal sub-period, the rationality of the determination of the abnormal sub-period can be ensured.

[0140] In order to ensure the rationality of the determination of historical operation data, based on the above embodiments, in the embodiments of the present application, an optional method for obtaining historical operation data is provided, such as Figure 6 As shown, the specific steps include:

[0141] S601, determining a historical detection period corresponding to the abnormal sub-period according to a time feature corresponding to the abnormal sub-period.

[0142] The so-called historical detection period is a period within the historical period that has the same time characteristics as the abnormal sub-period.

[0143] It is understandable that in order to improve the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period, in an optional implementation, the historical detection period corresponding to the abnormal sub-period can be determined from the historical period based on the time characteristics corresponding to the abnormal sub-period. Wherein, in the case of multiple historical detection periods, the historical detection period close to the abnormal sub-period in the chronological order is preferentially selected.

[0144] For example, when the abnormal sub-period is October 2024, the historical detection periods corresponding to the abnormal sub-period may be October 2023 and October 2022, among which October 2023 is preferentially selected as the historical detection period.

[0145] S602: Select historical operation data that meets the operation characteristics of the target operation and is in a normal operation state from each candidate operation data within the historical detection period.

[0146] The so-called candidate operation data refers to the operation data generated when each job is executed during the historical detection period.

[0147] It is understandable that since the types of jobs running on the intelligent computing network are rich and diverse, and each has different operating characteristics, when selecting historical operating data, it is necessary to select the operating data of other jobs similar to the target job during the historical detection period as the historical operating data.

[0148] In addition, since there are many reasons for job abnormalities, in the embodiment of the present application, only the operating data generated when the job is in a normal operating state is extracted as a forward reference to determine whether there is abnormal operating data in the abnormal sub-period.

[0149] In an optional implementation, the operation characteristics of the target operation under each preset dimension can be extracted; then, based on the operation characteristics, the candidate operation data that meets the operation characteristics and is in a normal operation state is screened out from the candidate operation data within the historical detection period as the historical operation data.

[0150] The preset dimensions may include but are not limited to user ID (user_id), group ID (group_id), queue (queue), number of nodes (node_count), number of CPUs (cpu_count), and application name (application).

[0151] For example, a data filtering template corresponding to the target job can be constructed based on the user ID, group ID, queue, number of nodes, number of CPUs and application name of the target job; then, the data filtering template corresponding to the target job is used to filter out the candidate operation data in a normal operation state from the candidate operation data within the historical detection period as the historical operation data.

[0152] In an embodiment of the present application, by determining the historical detection period based on the time feature dimension, and screening out the historical operation data in the normal operation state based on the operation characteristics of the target operation, the rationality and accuracy of the determination of the historical operation data can be ensured.

[0153] In order to ensure the accuracy of the abnormality detection result, based on the above embodiment, in the embodiment of the present application, an optional method for determining the abnormality detection result is provided, such as Figure 7 As shown, the specific steps include:

[0154] S701, determining the similarity between historical operation data and local operation data corresponding to an abnormal sub-period.

[0155] In an optional implementation, the historical operating data and the local operating data corresponding to the abnormal sub-period may be simultaneously input into a trained second similarity determination model, and the second similarity determination model outputs the similarity between the historical operating data and the local operating data based on the historical operating data, the local operating data and the model parameters.

[0156] In another optional implementation, in order to save time in tuning model parameters, the Copula-Based Outlier Detection (COPOD) algorithm based on statistical probability can be directly sampled to process the historical operation data and the local operation data corresponding to the abnormal sub-period, so as to obtain the similarity between the historical operation data and the local operation data. Among them, COPOD can be used to effectively model the dependency relationship between multiple random variables.

[0157] S702: Determine an anomaly detection result of the target job according to the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0158] In an optional implementation, since the historical operation data is data generated when the job is in a normal operating state, the abnormal detection result of the target job can be determined based on the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period, and the size relationship between the similarity threshold. Among them, the similarity threshold can be determined based on the historical experience of relevant technical personnel, or based on a large number of experimental results, which is not limited in this application.

[0159] Exemplarily, when the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period is greater than the similarity threshold, it can be determined that the abnormality detection result of the target job is that the job operation is normal; when the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period is less than or equal to the similarity threshold, it can be determined that the abnormality detection result of the target job is that the job operation is abnormal.

[0160] In the embodiment of the present application, by determining the abnormality detection result of the target job based on the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period, the accuracy of the abnormality detection result can be guaranteed.

[0161] Figure 8 This is a flow chart of a method for detecting abnormal operation of an intelligent computing network in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a method for detecting abnormal operation of an intelligent computing network. Figure 8 The specific implementation process is as follows:

[0162] S801, perform first-order difference processing on the original operation data of the target job in the intelligent computing power network during the period to be detected, and obtain the global operation data of the target job during the period to be detected.

[0163] S802, splitting the global operation data to obtain local operation data corresponding to each sub-period within the time period to be detected.

[0164] S803, determining the data change trend of the local operation data in each sub-period, and the data change trend of the local operation data in the neighboring sub-periods corresponding to each sub-period.

[0165] S804, determining a similarity score for each sub-period according to the trend similarity between the data change trend of each sub-period and the data change trend of the neighboring sub-period corresponding to each sub-period.

[0166] The neighbor sub-period is a sub-period adjacent to the sub-period in each sub-period, and the neighbor sub-period is located before or after the sub-period.

[0167] S805 , selecting an outlier similarity score from the similarity scores of each sub-period, and taking the sub-period corresponding to the outlier similarity score as an abnormal sub-period.

[0168] S806: Determine a historical detection period corresponding to the abnormal sub-period according to the time feature corresponding to the abnormal sub-period.

[0169] S807, selecting historical operation data that meets the operation characteristics of the target operation and is in a normal operation state from each candidate operation data within the historical detection period.

[0170] S808: Determine the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0171] S809: Determine an anomaly detection result of the target job according to the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0172] The specific process of the above S801-S809 can be found in the description of the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0173] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0174] Based on the same inventive concept, the embodiment of the present application also provides a device for detecting abnormal operation of an intelligent computing power network for realizing the method for detecting abnormal operation of an intelligent computing power network involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the device for detecting abnormal operation of an intelligent computing power network provided below can refer to the limitations of the method for detecting abnormal operation of an intelligent computing power network above, and will not be repeated here.

[0175] In an exemplary embodiment, Fig. 9 As shown, a job anomaly detection device 1 for an intelligent computing network is provided, comprising: a data splitting module 10, an anomaly determination module 20, a data acquisition module 30 and an anomaly detection module 40, wherein:

[0176] The data splitting module 10 is used to split the original operation data of the target job in the intelligent computing network within the period to be detected, and obtain the local operation data corresponding to each sub-period within the period to be detected;

[0177] The abnormality determination module 20 is used to determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods;

[0178] A data acquisition module 30, used to acquire historical operation data in a historical detection period corresponding to an abnormal sub-period;

[0179] The anomaly detection module 40 is used to determine an anomaly detection result of a target job according to the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0180] In an exemplary embodiment, the data splitting module 10 is specifically used for:

[0181] Perform first-order difference processing on the original operation data of the target job in the intelligent computing network during the period to be detected to obtain the global operation data of the target job during the period to be detected; split the global operation data to obtain the local operation data corresponding to each sub-period within the period to be detected.

[0182] In an exemplary embodiment, the abnormality determination module 20 includes:

[0183] A score determination unit, for determining, for each sub-period, a similarity score of the sub-period according to the similarity between the local operation data corresponding to the sub-period and the local operation data corresponding to the neighboring sub-period; wherein the neighboring sub-period is a sub-period adjacent to the sub-period in each sub-period, and the neighboring sub-period is located before or after the sub-period;

[0184] The abnormality determination unit is used to determine an abnormal sub-period from each sub-period according to the similarity score of each sub-period.

[0185] In an exemplary embodiment, the score determination unit is specifically configured to:

[0186] Determine the data change trend of the local operation data corresponding to the sub-period and the data change trend of the local operation data corresponding to the neighboring sub-period; determine the similarity score of the sub-period according to the trend similarity between the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period.

[0187] In an exemplary embodiment, the abnormality determination unit is specifically configured to:

[0188] From the similarity scores of each sub-period, an outlier similarity score is selected; and the sub-period corresponding to the outlier similarity score is taken as an abnormal sub-period.

[0189] In an exemplary embodiment, the data acquisition module 30 is specifically used for:

[0190] According to the time characteristics corresponding to the abnormal sub-period, the historical detection period corresponding to the abnormal sub-period is determined; from each candidate operation data within the historical detection period, the historical operation data that meets the operation characteristics of the target operation and is in a normal operation state is selected.

[0191] In an exemplary embodiment, the anomaly detection module 40 is specifically used to:

[0192] Determine the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period; and determine the abnormality detection result of the target job based on the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period.

[0193] Each module in the above-mentioned operation anomaly detection device for intelligent computing network can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0194] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store historical operation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting job anomalies for an intelligent computing power network is implemented.

[0195] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0196] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0197] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0198] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0199] It should be noted that the data involved in this application (including but not limited to historical operation data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0200] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting abnormal operation of an intelligent computing network, characterized in that: The method comprises: Splitting the original operation data of the target job in the intelligent computing network within the time period to be detected to obtain local operation data corresponding to each sub-time period within the time period to be detected; Determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods; Obtaining historical operation data within the historical detection period corresponding to the abnormal sub-period; An abnormality detection result of the target job is determined according to the historical operation data and the local operation data corresponding to the abnormal sub-period.

2. The method according to claim 1, characterized in that: The original operation data of the target job in the intelligent computing power network within the time period to be detected is split to obtain the local operation data corresponding to each sub-time period within the time period to be detected, including: Performing first-order difference processing on the original operation data of the target job in the intelligent computing power network during the period to be detected, to obtain the global operation data of the target job during the period to be detected; The global operation data is split to obtain local operation data corresponding to each sub-period within the time period to be detected.

3. The method according to claim 1, characterized in that The determining of the abnormal sub-period from each of the sub-periods according to the similarity between the local operation data corresponding to two adjacent sub-periods includes: For each sub-period, a similarity score of the sub-period is determined according to the similarity between the local operation data corresponding to the sub-period and the local operation data corresponding to the neighboring sub-period; wherein the neighboring sub-period is a sub-period adjacent to the sub-period in each sub-period, and the neighboring sub-period is located before or after the sub-period; According to the similarity scores of the sub-periods, abnormal sub-periods are determined from among the sub-periods.

4. The method according to claim 3, characterized in that The determining the similarity score of the sub-period according to the similarity between the local operation data corresponding to the sub-period and the local operation data corresponding to the neighboring sub-period includes: Determine a data change trend of the local operation data corresponding to the sub-period, and a data change trend of the local operation data corresponding to the neighbor sub-period; The similarity score of the sub-period is determined according to the trend similarity between the data change trend corresponding to the sub-period and the data change trend corresponding to the neighboring sub-period.

5. The method according to claim 3, characterized in that: The step of determining an abnormal sub-period from each sub-period according to the similarity score of each sub-period includes: From the similarity scores of each sub-period, select the outlier similarity score; The sub-period corresponding to the outlier similarity score is taken as the abnormal sub-period.

6. The method according to claim 1, characterized in that The acquiring of historical operation data in the historical detection period corresponding to the abnormal sub-period includes: Determine, according to the time feature corresponding to the abnormal sub-period, a historical detection period corresponding to the abnormal sub-period; From each candidate operation data within the historical detection period, historical operation data that meets the operation characteristics of the target operation and is in a normal operation state is selected.

7. The method according to claim 1, characterized in that The determining, according to the historical operation data and the local operation data corresponding to the abnormal sub-period, the abnormality detection result of the target job includes: Determining the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period; The abnormality detection result of the target job is determined according to the similarity between the historical operation data and the local operation data corresponding to the abnormal sub-period.

8. A device for detecting abnormal operation of an intelligent computing network, characterized in that: The device comprises: A data splitting module, used to split the original operation data of the target job in the intelligent computing network within the period to be detected, and obtain the local operation data corresponding to each sub-period within the period to be detected; an abnormality determination module, used to determine an abnormal sub-period from each sub-period according to the similarity between the local operation data corresponding to two adjacent sub-periods; A data acquisition module, used to acquire historical operation data in a historical detection period corresponding to the abnormal sub-period; The anomaly detection module is used to determine the anomaly detection result of the target job according to the historical operation data and the local operation data corresponding to the abnormal sub-period.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images