Big data anomaly detection processing method based on machine learning

Abstract

The invention discloses a big data anomaly detection processing method based on machine learning, and relates to the technical field of data processing.The method comprises the steps that task running logs of K big data tasks in a preset historical window are obtained, and K historical running log sequences are obtained; obtaining K concentrated abnormal feature set sequences; performing abnormal feature correlation iteration identification, determining K task abnormal coefficient sets of the K big data tasks, and configuring K abnormal detection paths according to the K task abnormal coefficient sets; according to the method and the device, the technical problem of low detection processing reliability caused by lack of deep analysis of implicit association conditions existing in historical operation of the big data tasks during big data anomaly detection in the prior art is solved; the technical effect of improving the anomaly detection precision is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data anomaly detection and processing method based on machine learning. Background Art

[0002] When executing big data tasks in a distributed computing environment, we often only judge whether there is an anomaly based on the current resource usage rate or task execution time of a single node, ignoring the potential anomaly correlation in the historical operation of the task. Although this method can discover some obvious anomalies, it may miss some potential anomalies because it fails to fully mine the hidden anomalies in the logs. This makes it difficult to provide accurate detection in anomaly detection of complex big data tasks.

[0003] The existing technology has the technical problem of lack of in-depth analysis of implicit correlations existing in the historical operation of big data tasks when detecting big data anomalies, resulting in low reliability of detection and processing. Summary of the invention

[0004] The present application provides a big data anomaly detection and processing method based on machine learning, which is used to solve the technical problem that in the prior art, big data anomaly detection lacks in-depth analysis of implicit correlations existing in the historical operation of big data tasks, resulting in low detection and processing reliability.

[0005] In view of the above problems, the present application provides a big data anomaly detection and processing method based on machine learning, the method comprising:

[0006] Obtain task running logs of K big data tasks in a preset historical window to obtain K historical running log sequences, where K is an integer greater than or equal to 1;

[0007] Traversing the K historical operation log sequences to perform concentrated extraction of abnormal features, and obtaining K concentrated abnormal feature set sequences, wherein each concentrated abnormal feature corresponds to an abnormal benchmark operation feature;

[0008] Iteratively identifying abnormal feature associations on the K historical operation log sequences and the K concentrated abnormal feature set sequences respectively, determining K task abnormality coefficient sets of the K big data tasks, and configuring K abnormality detection paths according to the K task abnormality coefficient sets;

[0009] Using the trained K anomaly detection paths to perform anomaly detection on the running conditions of the K big data tasks, to obtain K anomaly detection results;

[0010] Anomaly detection processing is performed on the K big data tasks according to the K anomaly detection results.

[0011] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0012] This application obtains K historical operation log sequences by obtaining task operation logs of K big data tasks in a preset historical window, where K is an integer greater than or equal to 1, and then traverses the K historical operation log sequences to extract abnormal features in a concentrated manner, and obtains K concentrated abnormal feature set sequences, where each concentrated abnormal feature corresponds to an abnormal baseline operation feature, and then performs abnormal feature association iterative identification on the K historical operation log sequences and the K concentrated abnormal feature set sequences respectively, determines the K task abnormal coefficient sets of the K big data tasks, and configures K abnormal detection paths according to the K task abnormal coefficient sets, and then uses the trained K abnormal detection paths to perform abnormal detection on the operation of the K big data tasks, obtains K abnormal detection results, and performs abnormal detection processing on the K big data tasks according to the K abnormal detection results. The technical effect of in-depth analysis of the historical operation of different big data tasks, differentiated abnormal detection, improved abnormal detection accuracy, and thus improved the reliability of abnormal processing is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A flowchart of a big data anomaly detection and processing method based on machine learning provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of a flow chart for determining a set of K task anomaly coefficients of K big data tasks in a big data anomaly detection and processing method based on machine learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The present application provides a big data anomaly detection and processing method based on machine learning, which is used to solve the technical problem that in the prior art, big data anomaly detection lacks in-depth analysis of implicit correlations existing in the historical operation of big data tasks, resulting in low detection and processing reliability.

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] It should be noted that the terms "including" and "having" are intended to cover non-exclusive inclusions. For example, a process, method, method, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0019] Examples, such as Figure 1 As shown, the present application provides a big data anomaly detection and processing method based on machine learning, the method comprising:

[0020] Step S100: Obtain task running logs of K big data tasks in a preset historical window to obtain K historical running log sequences, where K is an integer greater than or equal to 1;

[0021] In a possible embodiment, the preset historical window is to determine the time range (historical window) for extracting logs, and the window can be set according to factors such as the task operation cycle and the frequency of abnormal occurrence. Optionally, it can be the last hour, day, week, etc. Preferably, the window size directly affects the amount of log data obtained. The larger the window, the longer the historical log sequence, but an overly long window may introduce redundant information and increase processing complexity. Therefore, it is necessary to reasonably set the window length according to the task execution situation and actual needs.

[0022] The K big data tasks are tasks that use big data for processing and analysis, and may be data periodic collection tasks, data conversion tasks, etc. Optionally, the K big data tasks may be tasks that are executed simultaneously, or may be multiple tasks that are executed continuously. Each big data task has a unique task identifier, and the task's running log is located and obtained based on the task identifier during subsequent traversal.

[0023] In one embodiment, the task running logs of the K big data tasks are extracted from the local file system, distributed file system (such as HDFS), database or log management system (such as ELK, etc.) in the preset historical window to obtain the K historical running log sequences. Each task running log includes task identifier, task name, execution time, execution node, resource utilization data, task progress, task final status and other data. The K historical running log sequences reflect the running status of the K big data tasks in the preset historical window, achieving the goal of providing data support for subsequent reliable big data anomaly detection.

[0024] Step S200: traversing the K historical operation log sequences to perform concentrated extraction of abnormal features, and obtaining K concentrated abnormal feature set sequences, wherein each concentrated abnormal feature corresponds to an abnormal benchmark operation feature;

[0025] Further, the K historical operation log sequences are traversed to perform concentrated extraction of abnormal features to obtain K concentrated abnormal feature set sequences. Step S200 of the embodiment of the present application further includes:

[0026] Obtaining K benchmark operating feature sets and K operating feature tolerance interval sets of the K big data tasks;

[0027] Taking the K benchmark operation feature sets and the K operation feature tolerance interval sets as constraints, performing abnormal feature screening on the K historical operation log sequences to determine K abnormal feature cluster sequences, wherein each abnormal feature cluster includes multiple abnormal operation feature sets existing in a historical operation log;

[0028] Based on the K abnormal feature cluster sequences, concentrated abnormal feature screening is performed to determine the K concentrated abnormal feature set sequences.

[0029] Furthermore, the benchmark operation feature set includes task execution time, resource utilization, connection time and access failure rate.

[0030] In one possible embodiment, the K historical operation log sequences record the operation status of the K big data tasks within a preset historical window, including data that have abnormalities during operation. Therefore, by centrally analyzing the abnormal features in the K historical operation log sequences, the centralized abnormal features that best reflect the abnormal situation in each historical operation log are determined, thereby obtaining the K centralized abnormal feature set sequences. Each centralized abnormal feature corresponds to an abnormal baseline operation feature, that is, each centralized abnormal feature reflects the main abnormal situation of a baseline operation feature in a historical operation log where an abnormality occurs. The K centralized abnormal feature set sequences reflect the abnormal development of the K historical operation log sequences within the preset historical window.

[0031] In a possible embodiment, matching is performed according to the task types of the K big data tasks to obtain K benchmark operating feature sets and K operating feature tolerance interval sets of the K big data tasks. The benchmark operating feature set is a feature that must be detected during the operation of each big data task, including task execution time, resource utilization, connection time, and access failure rate. The K operating feature tolerance interval sets are the ranges that each benchmark operating feature that meets the operating requirements of the K big data tasks needs to meet.

[0032] Furthermore, with the K benchmark operation feature sets and the K operation feature tolerance interval sets as constraints, the data features that do not meet the constraints in the K historical operation log sequences are extracted to obtain K abnormal feature cluster sequences, wherein each abnormal feature cluster includes multiple abnormal operation feature sets existing in a historical operation log. By obtaining the K abnormal feature cluster sequences and performing concentrated screening of abnormal features, the K concentrated abnormal feature set sequences are determined. The technical effect of providing data support for the subsequent analysis of the prevalence of abnormal features in each historical operation log sequence is achieved.

[0033] Further, based on the K abnormal feature cluster sequences, the abnormal feature concentrated screening is performed to determine the K concentrated abnormal feature set sequences. In the embodiment of the present application, step S200 further includes:

[0034] Randomly extracting a first abnormal feature cluster from the K abnormal feature cluster sequences, and randomly extracting a first abnormal operation feature set from the first abnormal feature cluster;

[0035] Calculating the mean of the first abnormal operation feature set to obtain a first abnormal operation feature mean;

[0036] Taking the mean of the first abnormal operation feature as a starting point, screening is performed in the first abnormal operation feature set according to a preset centralized screening step length to determine a first-stage abnormal operation feature, wherein the first-stage abnormal operation feature is any first abnormal operation feature in the first abnormal operation feature set whose distance to the mean of the first abnormal operation feature is the preset centralized screening step length;

[0037] Determine whether the concentration density difference between the first-stage abnormal operation feature and the first abnormal operation feature mean is less than or equal to a preset difference; if not, when the concentration density of the first-stage abnormal operation feature is greater than the concentration density of the first abnormal operation feature mean, update the first-stage abnormal operation feature as the starting point, continue to screen in the first abnormal operation feature set until a preset number of screening times is met, take the starting point corresponding to the maximum concentration density in the screening as the target point, and take the target point as the first concentrated abnormal feature;

[0038] The K abnormal feature cluster sequences are subjected to abnormal feature concentrated screening according to a preset concentrated screening step length to determine the K concentrated abnormal feature set sequences.

[0039] Furthermore, if yes, the centralized screening is stopped, and the mean of the first abnormal operation feature is used as the first centralized abnormal feature.

[0040] In one embodiment, a first abnormal feature cluster is randomly extracted from the K abnormal feature cluster sequences, and a first abnormal operation feature set is randomly extracted from the first abnormal feature cluster. Then, the mean of the first abnormal operation feature set is calculated to obtain the first abnormal operation feature mean. The first abnormal operation feature mean reflects the general situation of an abnormal benchmark operation feature of the first abnormal operation feature set when considering the edge value.

[0041] Taking the mean of the first abnormal operation feature as the starting point, screening is performed in the first abnormal operation feature set according to a preset centralized screening step length to determine the first stage abnormal operation feature, wherein the preset centralized screening step length is a feature difference value of a single move when performing centralized screening, which is preset by a person skilled in the art. The first stage abnormal operation feature is any first abnormal operation feature in the first abnormal operation feature set whose distance to the mean of the first abnormal operation feature is the preset centralized screening step length.

[0042] Preferably, a mean screening area is constructed with the mean of the first abnormal operation feature as the center and the preset concentrated screening step as the radius, the number of first abnormal operation features in the mean screening area is counted, and the statistical result is divided by the area of ​​the mean screening area to obtain the concentration density of the first abnormal operation feature mean. Based on the same calculation principle, the concentration density of the abnormal operation features of the first stage is obtained.

[0043] Furthermore, it is determined whether the concentration density difference between the first-stage abnormal operation characteristics and the first abnormal operation characteristic mean is less than or equal to a preset difference. If not, it indicates that the density difference between the first-stage abnormal operation characteristics and the first abnormal operation characteristics distributed around the first abnormal operation characteristic mean is large. Therefore, the first abnormal operation characteristic mean cannot represent the general situation of the first abnormal operation characteristic set.

[0044] At this time, further judge the size of the concentration density of the first-stage abnormal operation characteristics and the concentration density of the mean of the first abnormal operation characteristics. When the concentration density of the first-stage abnormal operation characteristics is greater than the concentration density of the mean of the first abnormal operation characteristics, it indicates that the number of features clustered around the first-stage abnormal operation characteristics is greater than the number of features clustered around the mean of the first abnormal operation characteristics. The first-stage abnormal operation characteristics are updated as the starting point, and continue to screen in the first abnormal operation feature set until the preset screening times (the maximum screening times set by technicians in this field) are met. The starting point corresponding to the maximum concentration density in the screening is used as the target point, and the target point is used as the first concentrated abnormal feature.

[0045] In one possible embodiment, if yes, it indicates that the number of features clustered around the abnormal operation feature in the first stage is not much different from the number of features clustered around the mean of the first abnormal operation feature. At this time, the area with relatively dense distribution has been reached. Therefore, the centralized screening can be stopped, and the mean of the first abnormal operation feature is used as the first centralized abnormal feature. Furthermore, based on the same principle of obtaining the first centralized abnormal feature, the K abnormal feature cluster sequences are subjected to centralized screening of abnormal features according to the preset centralized screening step length to determine the K centralized abnormal feature set sequence. The technical effect of performing data processing on the K abnormal feature cluster sequences, reducing the amount of analysis data and improving data processing efficiency on the basis of ensuring data reliability is achieved.

[0046] Step S300: performing abnormal feature association iterative identification on the K historical operation log sequences and the K concentrated abnormal feature set sequences respectively, determining K task abnormal coefficient sets of the K big data tasks, and configuring K abnormality detection paths according to the K task abnormal coefficient sets;

[0047] Further, such as Figure 2 As shown, the K historical operation log sequences and the K concentrated abnormal feature set sequences are respectively subjected to abnormal feature association iterative identification to determine K task abnormal coefficient sets of the K big data tasks. Step S300 of the embodiment of the present application further includes:

[0048] Using the K concentrated abnormal feature set sequences as indexes, first-order keyword retrieval is performed in the K historical operation log sequences to obtain K first-order keyword group set sequences, wherein each first-order keyword group includes a plurality of keywords in the historical operation logs that describe the scenario of each concentrated abnormal feature;

[0049] Extracting the first K first-order keyword group sets in the first position in the K first-order keyword group set sequence, and sequentially performing associated keyword capture on the K first-order keyword group sets in a front-to-back order based on the K first-order keyword group sets to obtain K second-order associated keyword group sets;

[0050] The K sets of second-order associated keyword groups are analyzed using an abnormal coefficient identifier to determine the K sets of task abnormal coefficients.

[0051] Further, extract the K first-order keyword group sets that are located at the first position in the K first-order keyword group set sequence, and perform associated keyword capture on the K first-order keyword group sets in order from front to back based on the K first-order keyword group sets to obtain K second-order associated keyword group sets. Step S300 of the embodiment of the present application also includes:

[0052] Acquire the K second-order first-order keyword group sets located at the second position in the K first-order keyword group set sequence, perform association capture on the K second-order first-order keyword group sets using the K first-order keyword group sets as indexes, and add the capture results to K second-order association capture memory banks;

[0053] Re-acquire K third-order first-order keyword group sets located at the third position in the K first-order keyword group set sequence, perform association capture on the K third-order first-order keyword group sets using the K second-order association capture memory banks as indexes, and update the K second-order association capture memory banks according to the capture results;

[0054] After multiple association iterations and identifications, the last historical first-order keyword group in the sequence is associated with keyword capture, and K updated second-order association capture memory banks are obtained;

[0055] The K second-order associated keyword group sets are obtained according to the updated K second-order associated capture memory banks.

[0056] In one possible embodiment, since the big data task relies on hardware support and software support during execution, and with the increase in usage time and memory occupancy, the execution of the big data task has a certain before-and-after correlation. Therefore, by performing abnormal feature association iterative identification on the K historical operation log sequences and the K concentrated abnormal feature set sequences, the degree of abnormality in the execution of the K big data tasks can be determined, that is, the K task abnormality coefficient sets are obtained. Among them, each task abnormality coefficient corresponds to a benchmark operation feature. In other words, the K task abnormality coefficient sets reflect the abnormality of each benchmark operation feature of the K big data tasks in the historical operation. Then, the K task abnormality coefficient sets are comprehensively analyzed to determine the cycle of abnormal detection of the K big data tasks, and then the K abnormal detection paths are configured. The technical effect of comprehensively grasping the abnormal state of the big data task in the historical operation and providing the corresponding abnormal detection path for the subsequent abnormal detection that fits the actual situation of the big data task is achieved.

[0057] In one embodiment, the K concentrated abnormal feature set sequences are used as indexes, and first-order keyword searches are performed in the K historical operation log sequences to obtain K first-order keyword group set sequences, wherein each first-order keyword group includes multiple keywords for describing the scene of each concentrated abnormal feature in the historical operation log, such as error messages, warning information, or execution status, etc. Preferably, the corresponding description sentence is located according to the position of the concentrated abnormal feature in the historical operation log, and then multiple words used for scene description in the description sentence are extracted to obtain the first-order keyword group.

[0058] The K first-order keyword group sets that are at the first position in the K first-order keyword group set sequence are extracted, and the K first-order keyword group sets are used as the starting point of the associated iterative identification, and then the K first-order keyword group sets are sequentially captured with associated keywords in the order from front to back according to the K first-order keyword group sets, to obtain K second-order associated keyword group sets. Among them, the K second-order keyword group sets fully reflect the abnormal conditions of the K big data tasks.

[0059] Preferably, the K second-order first-order keyword group sets located in the second position in the K first-order keyword group set sequence are obtained, and the K second-order first-order keyword group sets are associated and captured with the K first-order first-order keyword group sets as indexes, and the captured results are added to K second-order associated capture memory banks. In other words, with the first-order first-order keyword group as the matching object, the keywords in the second-order first-order keyword group whose semantic similarity with the matching object meets the preset semantic similarity threshold are extracted and added to the corresponding second-order associated capture memory bank, so as to achieve the goal of cross-log iterative association recognition.

[0060] Based on the same principle, the K third-order first-order keyword group sets in the third position in the K first-order keyword group set sequence are obtained again, and the K third-order first-order keyword group sets are associated and captured with the K second-order associated capture memory banks as indexes, and the K second-order associated capture memory banks are updated according to the capture results, and after multiple associated iterative identifications, until the last historical first-order keyword group set in the sequence is associated with keyword capture, the updated K second-order associated capture memory banks are obtained, and the K second-order associated keyword group sets are obtained according to the updated K second-order associated capture memory banks. The technical effect of reducing the amount of analysis data and improving the extraction efficiency of abnormal analysis data for big data tasks is achieved on the basis of ensuring data reliability.

[0061] Furthermore, the K sets of second-order associated keyword groups are analyzed using an abnormal coefficient identifier to determine the K sets of task abnormal coefficients. Wherein, the abnormal coefficient identifier is a neural model constructed based on a feedforward neural network. By obtaining multiple sets of sample associated keyword groups and multiple sets of sample task abnormal coefficients as training data, supervised training is performed on the framework constructed based on the feedforward neural network, and the network parameters are updated according to the training output results during the training process until convergence, and the trained abnormal coefficient identifier is obtained. The K sets of second-order associated keyword groups are input into the abnormal coefficient identifier for analysis to obtain the K sets of task abnormal coefficients. The technical effect of intelligently analyzing and identifying abnormal situations of big data tasks is achieved.

[0062] Furthermore, step S300 of the embodiment of the present application further includes:

[0063] Traversing and calculating the mean of the K task abnormality coefficient sets to obtain the mean of the K task abnormality coefficients;

[0064] Respectively calculate the ratio of the mean values ​​of the abnormal coefficients of the K tasks to the sum of the mean values ​​of the abnormal coefficients of the K tasks, and multiply the difference between the ratio and 1 by the preset detection step length to obtain K detection step lengths;

[0065] K anomaly detection paths are configured based on the K detection step sizes.

[0066] In one possible embodiment, the mean of the K task anomaly coefficient sets is calculated respectively to obtain the mean of the K task anomaly coefficients. The mean of the K task anomaly coefficients reflects the average level of the K task anomaly coefficient sets. Further, the ratio of the mean of the K task anomaly coefficients to the sum of the mean of the K task anomaly coefficients is calculated respectively, and the difference between the ratio and 1 is multiplied by the preset detection step length to obtain K detection step lengths. The preset detection step length is a cycle for abnormality detection of big data tasks pre-set by a technician in this field. The K detection step lengths are detection cycles set according to the abnormal conditions of the K big data tasks. The larger the mean of the detection task anomaly coefficient, the higher the degree of abnormality, and the smaller the corresponding detection step length, and the more frequent the detection.

[0067] Obtain multiple sample task operation logs and multiple sample anomaly detection results as path training data, and divide the path training data into training set, validation set and test set according to a preset ratio. Optionally, the preset ratio is 5:3:2. The training set is used for model learning, the validation set is used for parameter adjustment, and the test set is used to finally evaluate the model performance. Then, the batch gradient descent method is adopted, and a small batch of samples is used each time to update the model parameters until the iterative training of all samples is completed. During the training, the validation set parameters are adjusted to adjust the learning rate, convolution kernel size, number of layers and other hyperparameters to ensure that the model can learn effectively and avoid overfitting. When the training converges, the initial anomaly detection path of the training is obtained. The K detection steps are respectively embedded in the initial anomaly detection path to obtain the K anomaly detection paths.

[0068] Step S400: using the trained K anomaly detection paths to perform anomaly detection on the running conditions of the K big data tasks to obtain K anomaly detection results;

[0069] Step S500: performing anomaly detection processing on the K big data tasks according to the K anomaly detection results.

[0070] In one embodiment, the K anomaly detection paths are used to perform anomaly detection on the operating data of the K big data tasks according to the K detection steps to obtain K anomaly detection results. The K anomaly detection results reflect the abnormal operation of the K big data tasks. Then, the K anomaly detection results are sent to the staff to perform anomaly detection processing on the K big data tasks. The technical effect of adapting to the operation of different big data tasks and performing anomaly detection on task operation is achieved.

[0071] In summary, the embodiments of the present application have at least the following technical effects:

[0072] The present application obtains K historical operation log sequences by obtaining task operation logs of K big data tasks in a preset historical window, wherein K is an integer greater than or equal to 1; traverses the K historical operation log sequences to extract abnormal features in a centralized manner, and obtains K centralized abnormal feature set sequences, wherein each centralized abnormal feature corresponds to an abnormal task operation indicator; performs abnormal feature association iteration identification on the K historical operation log sequences and the K centralized abnormal feature set sequences respectively, determines the K task abnormal coefficient sets of the K big data tasks, and configures K abnormal detection paths according to the K task abnormal coefficient sets; uses the trained K abnormal detection paths to perform abnormal detection on the operation of the K big data tasks, obtains K abnormal detection results, calls the abnormal processor to analyze the K abnormal detection results respectively, determines K processing schemes, and performs abnormal processing on the K big data tasks according to the K processing schemes. It achieves the technical effect of capturing features closely related to task execution abnormalities, and performing iterative association analysis, thereby ensuring the comprehensiveness of abnormal detection.

[0073] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0075] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A big data anomaly detection and processing method based on machine learning, characterized in that: The method comprises: Obtain task running logs of K big data tasks in a preset historical window to obtain K historical running log sequences, where K is an integer greater than or equal to 1; Traversing the K historical operation log sequences to perform concentrated extraction of abnormal features, and obtaining K concentrated abnormal feature set sequences, wherein each concentrated abnormal feature corresponds to an abnormal benchmark operation feature; Iteratively identifying abnormal feature associations on the K historical operation log sequences and the K concentrated abnormal feature set sequences respectively, determining K task abnormality coefficient sets of the K big data tasks, and configuring K abnormality detection paths according to the K task abnormality coefficient sets; Using the trained K anomaly detection paths to perform anomaly detection on the running conditions of the K big data tasks, to obtain K anomaly detection results; Anomaly detection processing is performed on the K big data tasks according to the K anomaly detection results.

2. The method for big data anomaly detection and processing based on machine learning according to claim 1, characterized in that: Traversing the K historical operation log sequences to perform concentrated extraction of abnormal features, and obtaining K concentrated abnormal feature set sequences, including: Obtaining K benchmark operating feature sets and K operating feature tolerance interval sets of the K big data tasks; Taking the K benchmark operation feature sets and the K operation feature tolerance interval sets as constraints, performing abnormal feature screening on the K historical operation log sequences to determine K abnormal feature cluster sequences, wherein each abnormal feature cluster includes multiple abnormal operation feature sets existing in a historical operation log; Based on the K abnormal feature cluster sequences, concentrated abnormal feature screening is performed to determine the K concentrated abnormal feature set sequences.

3. The method for big data anomaly detection based on machine learning as claimed in claim 2, characterized in that: Performing concentrated screening of abnormal features based on the K abnormal feature cluster sequences to determine the K concentrated abnormal feature set sequences includes: Randomly extracting a first abnormal feature cluster from the K abnormal feature cluster sequences, and randomly extracting a first abnormal operation feature set from the first abnormal feature cluster; Calculating the mean of the first abnormal operation feature set to obtain a first abnormal operation feature mean; Taking the mean of the first abnormal operation feature as a starting point, screening is performed in the first abnormal operation feature set according to a preset centralized screening step length to determine a first-stage abnormal operation feature, wherein the first-stage abnormal operation feature is any first abnormal operation feature in the first abnormal operation feature set whose distance to the mean of the first abnormal operation feature is the preset centralized screening step length; Determine whether the concentration density difference between the first-stage abnormal operation feature and the first abnormal operation feature mean is less than or equal to a preset difference; if not, when the concentration density of the first-stage abnormal operation feature is greater than the concentration density of the first abnormal operation feature mean, update the first-stage abnormal operation feature as the starting point, continue to screen in the first abnormal operation feature set until a preset number of screening times is met, take the starting point corresponding to the maximum concentration density in the screening as the target point, and take the target point as the first concentrated abnormal feature; The K abnormal feature cluster sequences are subjected to abnormal feature concentrated screening according to a preset concentrated screening step length to determine the K concentrated abnormal feature set sequences.

4. The method for big data anomaly detection and processing based on machine learning as claimed in claim 3, characterized in that: If yes, stop the centralized screening and take the mean of the first abnormal operation feature as the first centralized abnormal feature.

5. The method for big data anomaly detection based on machine learning as claimed in claim 2, characterized in that: The benchmark run feature set includes task execution time, resource utilization, connection time, and access failure rate.

6. The method for big data anomaly detection and processing based on machine learning according to claim 1, characterized in that: The K historical operation log sequences and the K concentrated abnormal feature set sequences are respectively subjected to abnormal feature association iterative identification to determine K task abnormality coefficient sets of the K big data tasks, including: Using the K concentrated abnormal feature set sequences as indexes, first-order keyword retrieval is performed in the K historical operation log sequences to obtain K first-order keyword group set sequences, wherein each first-order keyword group includes a plurality of keywords in the historical operation logs that describe the scenario of each concentrated abnormal feature; Extracting the first K first-order keyword group sets in the first position in the K first-order keyword group set sequence, and sequentially performing associated keyword capture on the K first-order keyword group sets in a front-to-back order based on the K first-order keyword group sets to obtain K second-order associated keyword group sets; The K sets of second-order associated keyword groups are analyzed using an abnormal coefficient identifier to determine the K sets of task abnormal coefficients.

7. The method for big data anomaly detection based on machine learning as claimed in claim 6, characterized in that: Extracting the first K first-order keyword group sets in the first position in the K first-order keyword group set sequence, and sequentially performing associated keyword capture on the K first-order keyword group sets in a front-to-back order based on the K first-order keyword group sets to obtain K second-order associated keyword group sets, including: Acquire the K second-order first-order keyword group sets located at the second position in the K first-order keyword group set sequence, perform association capture on the K second-order first-order keyword group sets using the K first-order keyword group sets as indexes, and add the capture results to K second-order association capture memory banks; Re-acquire K third-order first-order keyword group sets located at the third position in the K first-order keyword group set sequence, perform association capture on the K third-order first-order keyword group sets using the K second-order association capture memory banks as indexes, and update the K second-order association capture memory banks according to the capture results; After multiple association iterations and identifications, the last historical first-order keyword group in the sequence is associated with keyword capture, and K updated second-order association capture memory banks are obtained; The K second-order associated keyword group sets are obtained according to the updated K second-order associated capture memory banks.

8. The method for big data anomaly detection and processing based on machine learning according to claim 7, characterized in that: include: Traversing and calculating the mean of the K task abnormality coefficient sets to obtain the mean of the K task abnormality coefficients; Calculate the ratio of the mean values ​​of the abnormal coefficients of the K tasks to the sum of the mean values ​​of the abnormal coefficients of the K tasks respectively, multiply the difference between the ratio and 1 by the preset detection step length, and obtain K detection step lengths; K anomaly detection paths are configured based on the K detection step sizes.

Images