Job resource competition identification method and device, and electronic device

Abstract

The application discloses a job resource competition identification method and device and electronic equipment. The application first determines each periodic job with periodic characteristics in different jobs under a predetermined environment, and determines each job in which job execution time overlaps as a candidate job set. Then, the application obtains resource dependency logical relationship information corresponding to different jobs and periodic characteristics of different candidate jobs in the candidate job set, and identifies each target job in which resource competition exists in the candidate job set according to the resource dependency logical relationship information corresponding to different jobs and the periodic characteristics of different candidate jobs in the candidate job set. Thus, the application identifies periodic jobs, and realizes identification of jobs with resource competition risks according to the periodic characteristics of the periodic jobs and the resource dependency logical relationship information corresponding to different jobs, thereby providing a decision basis for improving platform job running stability and application experience.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of job scheduling and resource management, and particularly relates to a job resource competition identification method and device and an electronic device. BACKGROUND

[0002] When jobs are executed in a cloud computing environment or a container platform with resource competition, platform operation abnormalities are often caused by resource competition between different jobs, and even affect the normal execution of part of the time SLAs (service level agreement) jobs.

[0003] Therefore, identifying jobs with resource competition risks in a cloud computing environment or a container platform to provide a basis for decision-making for improving platform job operation stability and application experience has become an important research direction in the field. SUMMARY

[0004] To this end, the present application discloses the following technical solutions:

[0005] A job resource competition identification method, the method comprising:

[0006] determining each periodic job with periodic characteristics in different jobs in a predetermined environment;

[0007] determining each periodic job in which the job execution time overlaps, to obtain a candidate job set;

[0008] obtaining resource dependency logical relationship information corresponding to different jobs, and obtaining periodic characteristics of different candidate jobs in the candidate job set;

[0009] According to the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set, identifying each target job in the candidate job set with resource competition to obtain a job identification result with a resource competition risk.

[0010] Optionally, the determining each periodic job with periodic characteristics in different jobs in a predetermined environment comprises:

[0011] obtaining time node information of a job, the time node information comprising time corresponding to different execution progress of the job; the time node information is obtained by identifying predetermined annotation information or job configuration information, or by detecting a monitoring item related to the job;

[0012] According to the time node information of the job, a time series analysis method is used to determine whether the job has periodic characteristics, and if so, the job is determined to be a periodic job.

[0013] Optionally, the determining of the periodic jobs in which execution time of each of the periodic jobs overlaps to obtain a candidate job set comprises:

[0014] The execution process of each of the periodic jobs is divided into a plurality of time windows according to a predetermined step to obtain time slices of each of the periodic jobs;

[0015] The time slices of each of the periodic jobs are compared to determine the periodic jobs that are executed simultaneously in at least one same time period corresponding to the time slices as candidate jobs to obtain the candidate job set.

[0016] Optionally, the obtaining of the resource dependency logical relationship information corresponding to different jobs comprises:

[0017] The job calling relationship information between different jobs, the connection relationship information between different resources of the jobs and the dependency relationship information of the jobs on the resources are obtained as the resource dependency logical relationship information.

[0018] The obtaining of the periodicity characteristics of different candidate jobs in the candidate job set comprises:

[0019] The periodicity characteristics of different candidate jobs determined by using a time series analysis method are obtained.

[0020] Optionally, the identifying of the target jobs in which resource competition exists in the candidate job set according to the resource dependency logical relationship information and the periodicity characteristics of different candidate jobs in the candidate job set comprises:

[0021] A root cause analysis tool of platform running abnormity is constructed according to the resource dependency logical relationship information and the periodicity characteristics of different candidate jobs in the candidate job set.

[0022] Based on the root cause analysis tool, each of the target jobs in which resource competition exists in the candidate job set is determined according to a predetermined rule.

[0023] The root cause analysis tool comprises a logical relationship between different platform running abnormity events and job running events, job resource conflict events and / or job / resource index abnormity events in a logical causal relationship diagram manner.

[0024] Optionally, the constructing of the root cause analysis tool of platform running abnormity according to the resource dependency logical relationship information and the periodicity characteristics of different candidate jobs in the candidate job set comprises:

[0025] The resource dependency logical relationship information is parsed, and an index or factor in which potential resource competition relationship exists is identified; the index or factor is a resource-related index or factor or a job-related index or factor.

[0026] According to the job invocation relationship information, the connection relationship information between different resources, the dependency relationship information of the jobs on the resources, at least part of the information in the index or factor with potential resource competition relationship, and the abnormal information of the monitoring items related to the jobs in the predetermined monitoring range, a root cause analysis tool for platform running abnormality is preliminarily constructed;

[0027] According to the periodic characteristics of different candidate jobs in the candidate job set, abnormal detection and analysis before and after the execution of each candidate job are performed in time slices, and abnormal detection and analysis results are obtained.

[0028] According to the abnormal detection and analysis results, the root cause analysis tool is adjusted.

[0029] Optionally, the abnormal detection and analysis before and after the execution of each candidate job in time slices, and the abnormal detection and analysis results include:

[0030] Platform running abnormality detection and analysis and / or index abnormality detection and analysis of jobs and resources are performed in different time slices of each candidate job, and abnormal detection and analysis results are obtained.

[0031] The root cause analysis tool is adjusted according to the abnormal detection and analysis results, including:

[0032] The abnormal detection and analysis results are integrated into the root cause analysis tool.

[0033] Optionally, the above method further includes:

[0034] Before the start of the planned execution job, resource state detection is performed on the related resources required for the execution of the job, and in the case that the detected resource state meets the state condition, at least the start of the job is suspended.

[0035] And / or, according to the job identification result, job scheduling is performed in the predetermined environment to at least reduce resource competition between different jobs.

[0036] A job resource competition identification device, the device includes:

[0037] A first determination module is configured to determine each periodic job with periodic characteristics in different jobs in a predetermined environment.

[0038] A second determination module is configured to determine each periodic job in which the execution time of the jobs overlaps, and obtain a candidate job set.

[0039] obtain resource dependency logical relationship information corresponding to different jobs, and obtain periodicity characteristics of different candidate jobs in the candidate job set;

[0040] identify each target job in the candidate job set that has resource competition according to the resource dependency logical relationship information and the periodicity characteristics of different candidate jobs in the candidate job set, and obtain a job identification result that has a resource competition risk.

[0041] An electronic device comprises:

[0042] a memory for storing a computer instruction set;

[0043] a processor for implementing the job resource competition identification method according to any one of the above by executing the instruction set stored on the memory.

[0044] According to the above scheme, the job resource competition identification method, device and electronic device disclosed by the present application first determine each periodic job having periodicity characteristics in different jobs under a predetermined environment, and determine each job having execution time overlap in each periodic job as a candidate job set, then obtain resource dependency logical relationship information corresponding to different jobs and periodicity characteristics of different candidate jobs in the candidate job set, and identify each target job in the candidate job set that has resource competition according to the resource dependency logical relationship information corresponding to different jobs and the periodicity characteristics of different candidate jobs in the candidate job set. Thus, the present application identifies periodic jobs, and realizes job identification that has a resource competition risk according to the periodicity characteristics of periodic jobs and the resource dependency logical relationship information corresponding to different jobs, thereby providing a decision basis for improving platform job running stability and application experience. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0046] Figure 1 is a process flow chart of the job resource competition identification method provided by the present application;

[0047] Figure 2 is a detailed execution logic chart of the method disclosed by the present application;

[0048] Figure 3 is a construction process chart of the root cause analysis tool provided by the present application;

[0049] Figure 4 is another process flow diagram of the job resource competition identification method provided by the present application;

[0050] Figure 5 is a structural schematic diagram of the job resource competition identification device provided by the present application;

[0051] Figure 6 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0053] Performing a job under a cloud computing environment or a container platform with resource competition often leads to platform running abnormities due to resource competition between different jobs, and even affects the normal execution of part of the execution time SLAs jobs. The applicant has found that jobs with resource competition are generally divided into two types:

[0054] 1) a transient job, for example, a one-time analysis or model training activity started manually;

[0055] 2) a timing job, for example, a timing backup or timing data processing job. This type of job has a periodic characteristic, and if several timing jobs exist in resource competition, the impact on platform running is particularly great. Platform resource competition and platform running abnormities are also mainly caused by this type of job.

[0056] Based on this, the present application discloses a job resource competition identification method, device and electronic device, mainly aiming at periodic jobs under a cloud computing environment or a container platform, identifying jobs with resource competition risks, providing decision basis for improving platform job running stability and application experience. The job resource competition identification method disclosed by the present application can be applied to devices in a plurality of general or special computing device environments or configurations, for example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing service platforms (such as cloud computing platforms) including the above devices or devices, and the like.

[0057] Referring to Figure 1 The process flow of the job resource competition identification method provided by the present application includes:

[0058] Step 101, determining each periodic job with periodic characteristics in different jobs in a predetermined environment.

[0059] The predetermined environment can be, but is not limited to, a job execution environment such as a cloud computing platform or a container platform that can be used to execute jobs. Each periodic job with periodic characteristics can be a timed backup job, a timed data processing job, etc. in the environment such as the cloud computing platform or the container platform, and the timed data processing job includes, but is not limited to, jobs such as timed calculation, timed report, and timed model training.

[0060] Referring to Figure 2 , determining each periodic job with periodic characteristics in different jobs in a predetermined environment can be further implemented as:

[0061] 11) Obtain time node information of the job, the time node information including time corresponding to different execution progress of the job; the time node information being obtained by identifying predetermined annotation information or job configuration information, or by detecting a monitoring item related to the job.

[0062] Specifically, in an embodiment, the job has predetermined annotation information or job configuration information. The predetermined annotation information can be related time information manually annotated for the job or automatically annotated for the job by the system, and the job configuration information includes related time information pre-configured for the job when the job is planned. The time information annotated or configured for the job includes, but is not limited to, execution duration, planned start time and end time of the job, etc.

[0063] For the job with predetermined annotation information or job configuration information, the annotation information or configuration information of the job is directly identified, and the time node information of the job is obtained by identifying and analyzing the annotation information or the configuration information. For example, the start time, the end time, and the execution duration of the job in the annotated or configured time information are identified, and the median of the job time is analyzed according to the start time and the end time of the job, or the median of the job time is analyzed according to the start time and the execution duration of the job, etc. Correspondingly, the time node information including, but not limited to, the start time of the job, the median of the job time, and the end time of the job is obtained.

[0064] In an embodiment, the job lacks predetermined annotation information and job configuration information. In this case, the time node information of the job is identified by analyzing monitoring data of a monitoring item related to the job.

[0065] In implementation, the monitoring range and monitoring items of the job can be prepared in advance, the monitoring range includes but is not limited to the host range of the relevant host (such as the server node in the cloud platform environment) for deploying the job and the virtual machine range of the relevant virtual machine in the host for deploying the job, and the range can be represented by a series of set host information such as server node information and corresponding virtual machine information in the server node. The monitoring items include but are not limited to the CPU (Central Processing Unit) usage rate used by the job during execution, the IO (Input / Output) throughput of the dependent storage resource, etc. For reference, see Figure 2 The operation and maintenance history data such as monitoring, log, alarm, etc. of the service, resource, application, etc. related to the monitoring items in the monitoring range can be extracted from the corresponding database such as the monitoring and alarm database, log analysis system, etc. in the data preparation stage as the monitoring data.

[0066] Then, the abnormality detection activity is performed on the monitoring data of the monitoring items in the monitoring range, and the time node information such as the start or end or time median of the job is identified and analyzed through the abnormality detection of the monitoring data of the monitoring items.

[0067] Among them, the Level Shift abnormality detection method can be used, but is not limited to, to perform abnormality detection on the monitoring item data, if the detection result is marked as the level of the monitoring value is improved in the Level Shift abnormality detection, it generally indicates the start of the job, otherwise if the detection result is marked as the level of the monitoring value is reduced, it generally indicates the stop, pause or resource release of the job, but is not limited to this, and according to specific settings, when the detection result is marked as the level of the monitoring value is reduced in the Level Shift abnormality detection, it is identified as the start of the job, and when the detection result is marked as the level of the monitoring value is improved, it is identified as the stop, pause or resource release of the job. Then, according to the start, stop, pause or resource release events of the job, the start time, job time median and end time of the job and other time node information are analyzed.

[0068] 12) According to the time node information of the job, the time series analysis method is used to determine whether the job has periodic characteristics, if it has, it is determined that the job is a periodic job.

[0069] After obtaining the time node information of the job based on the predetermined marking information / job configuration information or through the monitoring items related to the job, the time series analysis method is used to identify the characteristics with periodic characteristics, for example, using the time series autocorrelation analysis method to obtain the job with typical periodic characteristics c1, c2, c3, etc., at the same time, the jobs without periodic characteristics are filtered out, and only the jobs with periodic characteristics are retained.

[0070] wherein c1, c2, c3 are respectively the periodic start time, end time, median time of the periodicity characteristics of the jobs.

[0071] Step 102, determine the jobs in the candidate job set in which the execution time of each periodic job overlaps.

[0072] After identifying each periodic job, the job duration, job start time, end time and other information embodied in the periodicity characteristics of the jobs are combined, and each periodic job is fragmented according to a predetermined step size, i.e. the execution process of each periodic job is divided into several time windows according to the step size, to obtain the time fragments of each periodic job in different periodic jobs, and the time fragments of each periodic job are compared to determine the periodic jobs that are executed simultaneously in at least one same time period corresponding to the time fragments (indicating that the execution time of these jobs overlaps), as candidate jobs, to obtain the candidate job set.

[0073] The candidate job set essentially embodies the potential job range in the job environment that has the possibility of resource competition / conflict (because the execution time of the jobs overlaps).

[0074] Step 103, obtain the resource dependency logical relationship information corresponding to different jobs, and obtain the periodicity characteristics of different candidate jobs in the candidate job set.

[0075] In this step, the job call relationship between different jobs, the connection relationship between different resources of the jobs, and the dependency relationship of the jobs on the resources and other information are specifically obtained as the resource dependency logical relationship information corresponding to different jobs. And the periodicity characteristics of different candidate jobs determined by the time series analysis method, such as typical periodicity characteristics c1, c2, c3, etc.

[0076] In implementation, see Figure 2 In the data preparation stage, the job scene call chain, the key indicators in the related resources, the related resource connection topology, the stress load distribution, the related labeling / configuration information and other information can be obtained from the cloud database, CMDB (Configuration Management Database), application database and other data storage devices in advance, to be used as the basis for analyzing the resource dependency logical relationship information corresponding to different jobs, to analyze the resource dependency logical relationship information corresponding to different jobs.

[0077] For example, the job call relationship between different jobs is analyzed according to the job scene call chain, the connection relationship between different resources of the jobs is analyzed according to the related resource connection topology and stress load distribution information (which nodes / virtual machines the jobs are dispatched to), and the dependency relationship between the jobs and the resources is analyzed according to the related labeling / configuration information.

[0078] Step 104, according to the resource dependency logical relationship information corresponding to different jobs and the periodic characteristics of different candidate jobs in the candidate job set, identifying each target job in the candidate job set that exists resource competition, obtaining the job identification result that exists resource competition risk.

[0079] Then, further according to the resource dependency logical relationship information corresponding to different jobs and the periodic characteristics of different candidate jobs in the candidate job set, a root cause analysis tool for platform running abnormity is constructed; and based on the root cause analysis tool, each target job in the candidate job set that exists resource competition is determined according to a predetermined rule.

[0080] Among them, the root cause analysis tool constructed includes the logical relationship between different platform running abnormity events and job running events, job resource conflict events and / or job / resource index abnormity events organized in a logical causal relationship diagram manner.

[0081] Specifically, referring to Figure 3 , the process of constructing the root cause analysis tool can be implemented as:

[0082] Step 301, parsing the resource dependency logical relationship information corresponding to different jobs, and identifying the indexes or factors that exist potential resource competition relationship.

[0083] Specifically, according to the dependency relationship between different jobs and resources analyzed based on relevant annotation / configuration information, and combining the key indexes in the relevant resources, the main indexes and factors that may produce resource competition can be identified. The indexes or factors here are resource-related indexes or factors, or job-related indexes or factors. For example, a database cluster job is sensitive to indexes such as storage IOPS (Input / Output Operations Per Second, the number of read / write operations per second), delay, throughput, etc., and may produce resource competition with other jobs in these indexes.

[0084] Step 302, according to the job invocation relationship information, the connection relationship information between different resources, the dependency relationship of jobs on resources, and at least part of the indexes or factors that exist potential resource competition relationship, and the abnormal information of the monitoring items related to the jobs in the predetermined monitoring range, a root cause analysis tool for platform running abnormity is preliminarily constructed.

[0085] The predetermined monitoring range and the monitoring items can be referred to the above description.

[0086] The step specifically generates a root cause analysis tool under the resource dependency topology according to at least part of the information of the job calling relationship information, the connection relationship information between different resources, the dependency relationship information of the job on the resources, and the indicators or factors with potential resource competition relationship, and the abnormal information of the monitoring items related to the job in the predetermined monitoring range during the job execution, such as CPU occupancy, IO throughput, database performance, storage performance, etc., so as to realize qualitative judgment in the occurrence of some performance or indicator abnormality.

[0087] For example, the database performance decreases, and at the same time, the storage performance decreases, and these performance decreases are caused by the database job or several jobs competing for resources. Such judgment logic will be embodied in the root cause analysis tool, so as to qualitatively judge the cause of the performance or indicator abnormality based on the root cause analysis tool, and correspondingly identify the jobs with resource competition.

[0088] Typical root cause analysis tools can perform top-down deductive analysis by fault tree analysis (FTA), and infer the problem by combining logical gates and basic events.

[0089] Step 303, in combination with the periodic characteristics of different candidate jobs in the candidate job set, the abnormal detection and analysis before and after the execution of each candidate job are performed according to time slicing, and the abnormal detection and analysis results are obtained.

[0090] In addition, in combination with the periodic characteristics (such as start time, end time, etc.) of different candidate jobs in the candidate job set, the abnormal detection and analysis before and after the execution of each candidate job are performed according to time slicing on the associated indicators and log data of each candidate job. For example, in combination with the resource dependency relationship according to time slicing, whether the performance of the SAN (Storage Area Network) storage device dependent on the node of the job is abnormal is identified, whether the main indicator change of other nodes connected to the SAN storage device except the job node before and after the job is abnormal, etc. For example, in a typical scenario, if some nodes present a large IO throughput during the execution of the job, the job may be affected, and there is potential resource competition.

[0091] The step 303 can be further implemented as:

[0092] 21) According to the periodic characteristics of the job, the jobs corresponding to each node in the monitoring range are divided into a plurality of time windows according to a predetermined step length, a series of time slices corresponding to the jobs are obtained, and the abnormal detection and analysis are performed on the associated indicator data, log data, etc. in each time slice.

[0093] In implementation, in combination with reference to Figure 2The operation-related service, resource, application, and the like in a certain data range can be extracted from a corresponding database, such as a monitoring and alarm database, a log analysis system, and the like, in advance in a data preparation stage, so as to be used for abnormality detection and analysis. The extracted data range can be a certain historical time period, for example, 3 days, 15 days, or 30 days of historical data before detection and analysis of an activity.

[0094] Optionally, abnormality detection and analysis can be performed on the corresponding associated index data, log data, and the like under each time slice in a sliding time window manner, wherein each time window is slid by 1 / N length of the time window (N is an integer greater than 1), so as to ensure that the abnormality in the time slice is fully captured and abnormal events are not missed due to crossing of the time slice.

[0095] The abnormality in the time window can be detected according to a predetermined abnormality detection algorithm and in combination with a set related threshold, and the abnormality detection of the data under each time slice is realized. The abnormality detection and analysis include, but are not limited to, platform running abnormality detection and analysis, job and resource index abnormality detection and analysis, and the like under different time slices of each candidate job.

[0096] For example, according to a predetermined abnormality detection algorithm and in combination with a set related threshold, data form abnormality or fluctuation abnormality of the data in the time window is detected, such as assuming that the CPU occupancy rate fluctuates between 10% and 50% in a certain period of time, and the CPU occupancy rate suddenly increases to 90% (exceeding a set occupancy rate threshold) from t time point, the data form abnormality or fluctuation abnormality is identified, and the CPU occupancy rate abnormality is identified accordingly.

[0097] 22) The time slice data of different jobs in each node is time-aligned, a specific abnormal event (such as a platform abnormal event of database performance degradation, storage performance degradation, and the like) is marked if a certain time slice is abnormal, and a series of time slices after time alignment are analyzed under a required logical gate (such as a sequential logical gate) according to a sliding window based on a preset step, and the abnormality detection and analysis result is obtained accordingly.

[0098] Specifically, for example, for an occurrence of an abnormal event, the reason for the occurrence of the event is analyzed layer by layer from top to bottom according to the sequential logical gate, such as a certain index abnormality, and the next layer of reasons for the index abnormality is analyzed, such as which different jobs have resource competition in which aspects.

[0099] Step 304: Adjusting the root cause analysis tool according to the abnormality detection and analysis result.

[0100] Afterwards, the result of the time-slicing based abnormality detection and analysis is further integrated into the root cause analysis tool constructed in step 302, and the tool is adjusted so that it can reflect the causal logical relationship between different platform running abnormality events and job running events, job resource conflict events and / or job / resource index abnormality events in the form of a logical causal relationship diagram, such as an FTA fault tree.

[0101] The finally constructed root cause analysis tool includes a series of events organized by logical gates, such as top events, intermediate events and bottom events, which reflect the causes of the top events from top to bottom through the connection relationship between different events and corresponding types of logical gates, so that for each top event, the corresponding bottom event can be finally located as the root cause of the top event through the logical gate, wherein the logical gate includes but is not limited to AND gate, XOR gate, NOT gate, etc., sequential gate, etc., the top event includes but is not limited to database performance abnormality, storage performance abnormality, etc. Platform abnormality, intermediate events include but are not limited to some factors affecting, some index abnormalities, etc., and bottom events include but are not limited to some resource conflicts, some job execution, etc., so as to reflect the logical relationship between different platform running abnormality events and job running events, job resource conflict events and / or job / resource index abnormality events.

[0102] On this basis, the resource conflicts prone to cause platform abnormality and the jobs in the candidate job set that will cause these resource conflicts can be further determined based on the top events, intermediate events, bottom events included in the root cause analysis tool and the logical gates used to connect events of various types, to obtain the job identification result that exists resource competition risk.

[0103] Optionally, in implementation, as shown in Figure 2 , the job identification result that exists resource competition risk obtained based on the root cause analysis tool can also be displayed to engineers and other relevant personnel through a visualization method, and combined with the manual labeling of resource conflict information or the judgment confirmation result of whether there is resource conflict, to obtain the final periodic job that exists resource competition risk.

[0104] In addition, optionally, as shown in Figure 2 , the identification result (which jobs exist resource competition, and what resource competition will cause which platform running abnormality, etc.) and the identification basis (such as the logical analysis process that the cause of the platform running abnormality is which jobs existing resource competition in which aspects) can also be processed and tailored according to the business scenario to generate information forms matching the business scenario, and sent to the relevant receiving party in the form of notification, message, email, report, etc. Or persisted to a storage device.

[0105] The above business scenario can be, but is not limited to, alarm, generating a report or integrating the identification result into other related processes (such as a job scheduling process), etc.

[0106] Subsequently, according to the identification result of the jobs with resource competition relationship, job scheduling can be performed in a cloud computing platform or a container platform, etc. to avoid simultaneous execution of each job or at least part of the jobs with resource competition relationship, so as to reduce resource competition between different jobs and avoid occurrence of platform running abnormity.

[0107] From the above solution, it can be seen that the method of the embodiment first determines each periodic job with periodic characteristics in different jobs in a predetermined environment, and determines each job with overlapping job execution time in each periodic job as a candidate job set. Then, the resource dependency logical relationship information corresponding to different jobs and the periodic characteristics of different candidate jobs in the candidate job set are obtained, and each target job with resource competition in the candidate job set is identified according to the resource dependency logical relationship information corresponding to different jobs and the periodic characteristics of different candidate jobs in the candidate job set. Thus, the application identifies periodic jobs, and realizes identification of jobs with resource competition risk according to the periodic characteristics of periodic jobs and the resource dependency logical relationship information corresponding to different jobs, thereby providing a decision basis for improving platform job running stability and application experience.

[0108] In an embodiment, optionally, the job resource competition identification method disclosed by the application can further include the following processing after the jobs with resource competition risk in the candidate job set are identified according to the root cause analysis tool:

[0109] The numerical results in the abnormality detection data are converted into event data or state data with operation and maintenance significance, and the identification result of the jobs with resource competition risk obtained according to the root cause analysis tool is verified according to the event data or state data with operation and maintenance significance.

[0110] Specifically, for the abnormality detection data (such as specific values of CPU occupancy rate, IO throughput, etc.) in step 22) in step 303, the numerical results in the abnormality detection data can be converted into event data or state data with operation and maintenance significance, such as CPU abnormality, IO throughput abnormality, etc. by statistical methods or manual calibration of static threshold, etc. Further, based on the converted event data or state data with operation and maintenance significance, the result determined based on the root cause analysis tool is verified and confirmed to verify the correctness of the result of the jobs with competition relationship identified based on the root cause analysis tool, and the root cause analysis tool and / or the identification result are corrected in the case of abnormality.

[0111] The embodiment converts the numerical results in the abnormality detection data into event data or state data with operation and maintenance significance, verifies the identification result of the job with resource competition risk obtained based on the root cause analysis tool according to the converted event data or state data with operation and maintenance significance, and further improves the accuracy of the identification result of the job with resource competition risk, which can provide more accurate and valuable decision basis for improving the platform job operation stability and application experience.

[0112] In an embodiment, referring to Figure 4 The flowchart of the provided job resource competition identification method, the job resource competition identification method disclosed in the application can further include the following processing:

[0113] In step 105, before starting the planned job, the resource state of the related resource required by the job is detected, and if the detected resource state meets the state condition, the starting of the job is at least suspended.

[0114] The state condition is a condition capable of representing the resource state abnormality of the related resource required by the job, such as the value of the resource index being outside the set threshold range.

[0115] The embodiment mainly aims at a one-off job, and performs the related resource state detection processing.

[0116] Specifically, before starting the planned one-off job, the state of the related resource required by the job in a recent predetermined time window is detected, for example, the state of the key computing, network, and storage resource in the last 30 minutes or 1 hour before the starting of the job is detected, if the resource index of the key computing, network, and storage resource required by the job appears abnormal in the time window, such as the value of the computing resource index being outside the threshold range, the job activity is suspended.

[0117] In addition, a notification message can be sent to the administrator and other related personnel for manual intervention, or the resource state of the related resource required by the job is continuously detected in real time / periodically, and the job is started when the resource returns to the normal state, such as the value of the computing resource index returning to the threshold range half an hour after the planned starting time.

[0118] The embodiment realizes the resource competition risk identification of the one-off job, which is complementary to the resource competition risk identification of the periodic job in the above method embodiment, can effectively avoid the further intensification of the resource competition caused by the job activity, and improves the overall reliability of the cloud platform and other job environments.

[0119] Corresponding to the above-mentioned job resource competition identification method, the embodiment of the application further discloses a job resource competition identification device, the component structure of the device is as shown in Figure 5 Specifically includes:

[0120] The first determination module 501 is configured to determine each periodic job with a periodic characteristic in different jobs in a predetermined environment.

[0121] The second determination module 502 is configured to determine each periodic job in which the execution time of the jobs overlaps, to obtain a candidate job set.

[0122] The acquisition module 503 is configured to acquire resource dependency logical relationship information corresponding to different jobs, and to acquire the periodic characteristic of different candidate jobs in the candidate job set.

[0123] The identification module 504 is configured to identify each target job in the candidate job set that has resource competition according to the resource dependency logical relationship information and the periodic characteristic of different candidate jobs in the candidate job set, to obtain a job identification result that has a resource competition risk.

[0124] In an embodiment, the first determination module 501 is specifically configured to:

[0125] Acquire time node information of the job, the time node information including a time corresponding to different execution progress of the job; the time node information is obtained by identifying predetermined annotation information or job configuration information, or by detecting a monitoring item related to the job;

[0126] According to the time node information of the job, determine whether the job has a periodic characteristic by using a time series analysis method, and if so, determine that the job is a periodic job.

[0127] In an embodiment, the second determination module 502 is specifically configured to:

[0128] Divide the execution process of each periodic job into a plurality of time windows according to a predetermined step size, to obtain a time slice of each periodic job in different periodic jobs;

[0129] Compare the time slices of each periodic job, to determine each periodic job that is executed simultaneously in at least one same time period corresponding to the time slices, as a candidate job, to obtain a candidate job set.

[0130] In an embodiment, when acquiring the resource dependency logical relationship information corresponding to different jobs, the acquisition module 503 is specifically configured to: acquire job call relationship information between different jobs, connection relationship information between different resources of the job, and dependency relationship information of the job on the resources, as the resource dependency logical relationship information.

[0131] The acquisition module 503 is configured to acquire the periodic characteristics of different candidate jobs in the candidate job set by using the time series analysis method.

[0132] In an embodiment, the identification module 504 is configured to:

[0133] According to the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set, a root cause analysis tool for platform running abnormalities is constructed.

[0134] Based on the root cause analysis tool, target jobs in the candidate job set that have resource competition are determined according to a predetermined rule.

[0135] The root cause analysis tool includes logical relationships between different platform running abnormal events and job running events, job resource conflict events, and / or job / resource index abnormal events in a logical causal relationship diagram.

[0136] In an embodiment, the identification module 504 is configured to:

[0137] The resource dependency logical relationship information is analyzed, and indexes or factors with potential resource competition relationships are identified. The indexes or factors are resource-related indexes or factors, or job-related indexes or factors.

[0138] According to the job calling relationship information, the connection relationship information between different resources, the dependency relationship information of jobs on resources, at least part of the indexes or factors with potential resource competition relationships, and abnormal information of monitoring items related to jobs within a predetermined monitoring range, a root cause analysis tool for platform running abnormalities is preliminarily constructed.

[0139] In combination with the periodic characteristics of different candidate jobs in the candidate job set, abnormal detection and analysis before and after the execution of each candidate job are performed according to time slicing, and abnormal detection and analysis results are obtained.

[0140] According to the abnormal detection and analysis results, the root cause analysis tool is adjusted.

[0141] In an embodiment, the identification module 504 is configured to, when performing abnormal detection and analysis before and after the execution of each candidate job according to time slicing, perform platform running abnormal detection and analysis and / or job and resource index abnormal detection and analysis under different time slicing of each candidate job, and obtain abnormal detection and analysis results.

[0142] The identification module 504 is configured to integrate the abnormality detection and analysis result into the root cause analysis tool when adjusting the root cause analysis tool according to the abnormality detection and analysis result.

[0143] In an embodiment, the apparatus further includes a detection module and / or a scheduling processing module.

[0144] The detection module is configured to perform resource state detection on the related resources required by the job before the job is started, and at least suspend the start of the job when the detected resource state meets the state condition.

[0145] The scheduling processing module is configured to perform job scheduling in the predetermined environment according to the job identification result, so as to at least reduce the resource competition between different jobs.

[0146] For the job resource competition identification apparatus disclosed in the embodiments of the present application, since it corresponds to the job resource competition identification method disclosed in the above method embodiments, the description is relatively simple, and the related similarities can be referred to the description of the corresponding method embodiments, which will not be described in detail here.

[0147] The embodiments of the present application further disclose an electronic device, which can be but is not limited to a server device in a cloud computing environment, and the constituent structure of the electronic device is as shown in Figure 6 The electronic device specifically includes:

[0148] The memory 601 is configured to store a computer instruction set.

[0149] The computer instruction set can be implemented in the form of a computer program.

[0150] The processor 602 is configured to implement the job resource competition identification method disclosed in any of the above method embodiments by executing the computer instruction set.

[0151] The processor 602 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, etc.

[0152] In addition, the electronic device can further include a communication interface, a communication bus and other constituent parts. The memory, the processor and the communication interface complete mutual communication through the communication bus.

[0153] The communication interface is used for communication between the electronic device and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc., which can be divided into an address bus, a data bus, a control bus, etc.

[0154] It should be noted that each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be mutually referred to.

[0155] For the convenience of description, the above system or device is described as various modules or units respectively described in terms of functions. Of course, in the implementation of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0156] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments of the present application.

[0157] Finally, it should be noted that in this document, relational terms such as first and second and third and fourth, and the like can merely be used to distinguish one entity or action from another, and do not necessarily require or imply that any such order or numbering exists between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0158] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for identifying job resource contention, the method comprising: Identify the periodic operations with periodic characteristics in different operations under a predetermined environment; Identify the periodic jobs whose execution times overlap among the various periodic jobs to obtain a candidate job set; Obtain the resource dependency logic relationship information corresponding to different jobs, and obtain the periodic characteristics of different candidate jobs in the candidate job set; Based on the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set, identify each target job in the candidate job set that has resource competition, and obtain the job identification result with resource competition risk; The step of identifying target jobs with resource contention in the candidate job set based on the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set includes: Based on the resource dependency logic relationship information and the periodic characteristics of different candidate jobs in the candidate job set, a root cause analysis tool for platform operation anomalies is constructed. Based on the root cause analysis tool, each target job in the candidate job set that has resource competition is determined according to predetermined rules; The root cause analysis tool includes logical relationships between different platform operational anomalies and job operation events, job resource conflict events, and / or job / resource indicator anomalies, organized in the form of a logical cause-and-effect diagram. The step of constructing a root cause analysis tool for platform operation anomalies based on the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set includes: The resource dependency logic relationship information is analyzed to identify indicators or factors that may have potential resource competition relationships; the indicators or factors are resource-related indicators or factors, or operation-related indicators or factors. Based on job call relationship information, connection relationship information between different resources, job dependency relationship information, and at least some of the information in the indicators or factors that indicate potential resource competition, as well as the abnormal information of monitoring items related to jobs within the predetermined monitoring scope, a root cause analysis tool for platform operation abnormalities is initially constructed. Based on the periodic characteristics of different candidate jobs in the candidate job set, anomaly detection and analysis are performed on each candidate job before and after execution by dividing it into time slices, and the anomaly detection and analysis results are obtained. Based on the anomaly detection and analysis results, adjust the root cause analysis tool.

2. The method according to claim 1, wherein determining each periodic operation with periodic characteristics in different operations under a predetermined environment includes: Obtain the time node information of the task, which includes the time corresponding to the task at different execution progresses; The time node information is obtained by identifying predetermined annotation information or job configuration information, or by detecting monitoring items related to the job. Based on the time node information of the task, time series analysis is used to determine whether the task has periodic characteristics. If it does, the task is determined to be a periodic task.

3. The method according to claim 1, wherein determining the periodic jobs whose execution times overlap among the periodic jobs to obtain a candidate job set includes: The execution process of each periodic task is divided into several time windows according to a predetermined step size, so as to obtain the time slice of each periodic task in different periodic tasks. By comparing the time slices of each periodic job, each periodic job that is executed simultaneously within at least one time period corresponding to the time slice is identified as a candidate job, thus obtaining a candidate job set.

4. The method according to claim 3, wherein obtaining the resource dependency logical relationship information corresponding to different jobs includes: Obtain job call relationship information between different jobs, connection relationship information between different resources of jobs, and dependency relationship information of jobs on resources, as the resource dependency logical relationship information; The step of obtaining the periodic characteristics of different candidate jobs in the candidate job set includes: Obtain the periodic characteristics of different candidate jobs identified using time series analysis.

5. The method according to claim 1, wherein the anomaly detection and analysis of each candidate job before and after job execution by dividing the job into time slices to obtain the anomaly detection and analysis results includes: Under different time segments of each candidate job, perform platform operation anomaly detection and analysis and / or job and resource indicator anomaly detection and analysis to obtain anomaly detection and analysis results; The step of adjusting the root cause analysis tool based on the anomaly detection and analysis results includes: The anomaly detection and analysis results are integrated into the root cause analysis tool.

6. The method according to claim 1, further comprising: Before a planned job is started, the resource status of the relevant resources required for job execution is checked, and if the detected resource status meets the status conditions, the start of the job is at least paused. And / or, based on the job identification results, job scheduling is performed in the predetermined environment to at least reduce resource competition between different jobs.

7. A task resource contention identification device, the device comprising: The first determining module is used to determine each periodic operation with periodic characteristics in different operations under a predetermined environment; The second determining module is used to determine each periodic job whose execution time overlaps among the various periodic jobs, and to obtain a candidate job set; The acquisition module is used to acquire resource dependency logic relationship information corresponding to different jobs, and to acquire the periodic characteristics of different candidate jobs in the candidate job set; The identification module is used to identify each target job in the candidate job set that has resource competition based on the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set, and to obtain the job identification result with resource competition risk. The identification module, based on the resource dependency logic information and the periodic characteristics of different candidate jobs in the candidate job set, identifies each target job in the candidate job set that has resource competition, specifically for: Based on the resource dependency logic relationship information and the periodic characteristics of different candidate jobs in the candidate job set, a root cause analysis tool for platform operation anomalies is constructed. Based on the root cause analysis tool, each target job in the candidate job set that has resource competition is determined according to predetermined rules; The root cause analysis tool includes logical relationships between different platform operational anomalies and job operation events, job resource conflict events, and / or job / resource indicator anomalies, organized in the form of a logical cause-and-effect diagram. The step of constructing a root cause analysis tool for platform operation anomalies based on the resource dependency logical relationship information and the periodic characteristics of different candidate jobs in the candidate job set includes: The resource dependency logic relationship information is analyzed to identify indicators or factors that may have potential resource competition relationships; the indicators or factors are resource-related indicators or factors, or operation-related indicators or factors. Based on job call relationship information, connection relationship information between different resources, job dependency relationship information, and at least some of the information in the indicators or factors that indicate potential resource competition, as well as the abnormal information of monitoring items related to jobs within the predetermined monitoring scope, a root cause analysis tool for platform operation abnormalities is initially constructed. Based on the periodic characteristics of different candidate jobs in the candidate job set, anomaly detection and analysis are performed on each candidate job before and after execution by dividing it into time slices, and the anomaly detection and analysis results are obtained. Based on the anomaly detection and analysis results, adjust the root cause analysis tool.

8. An electronic device, comprising: Memory is used to store computer instruction sets; A processor is configured to implement the job resource contention identification method as described in any one of claims 1-6 by executing a set of instructions stored in memory.

Images