Batch-stream integrated indicator data anomaly detection method and device

Abstract

The present invention discloses a method and device for detecting abnormality of batch-stream integrated indicator data, and relates to the technical field of big data processing. The method comprises: screening and processing the indicator data outputted by the data source kafka to obtain the processed indicator data; inputting the processed indicator data into the batch-stream integrated kafka; writing the processed indicator data outputted by the batch-stream integrated kafka into the data stock table corresponding to the processed indicator data; performing model training according to the indicator data in the data stock table and the preset indicator data prediction algorithm to train an indicator data prediction model; performing abnormality detection on the processed indicator data outputted by the batch-stream integrated kafka according to the indicator data prediction value outputted by the indicator data prediction model. The present invention is helpful to improve the effect of abnormality detection of indicator data.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method and device for detecting abnormality of batch and stream integrated indicator data. Background Art

[0002] One of the characteristics of the Internet era is the amount of big data. Big data batch processing and streaming analysis processing tasks and needs can be seen everywhere. Take data center operation and maintenance data processing as an example. With the accelerated development of business, the amount of data generated by various physical devices and business application indicators is growing exponentially. In order to improve the efficiency and quality of operation and maintenance, we need to process streaming data in quasi-real time to timely discover abnormal data in indicator data. On the other hand, we need to batch process historical data to train the parameters of the algorithm used for anomaly identification to improve the accuracy of anomaly detection. These all involve the batch processing and stream processing technology of big data. At present, big data analysis platforms generally split stream data processing and batch data processing into two different independent architectures, that is, batch processing for training anomaly identification algorithms and stream processing for quasi-real-time identification of abnormal indicator data are performed independently in two separate architectures. This makes the current development and operation and maintenance of indicator data anomaly detection cumbersome and complex, and it is difficult to cope with the changing needs of operation and maintenance big data scenarios. Summary of the invention

[0003] In order to solve the technical problems in the above-mentioned background technology, the present invention proposes a batch-stream integrated indicator data anomaly detection method and device.

[0004] In order to achieve the above object, according to one aspect of the present invention, a method for detecting abnormality of batch-stream integrated indicator data is provided, the method comprising:

[0005] The indicator data output by the data source kafka is filtered according to the preset data filtering conditions, and the filtered indicator data is processed according to the preset data processing method to obtain the processed indicator data;

[0006] Input the processed indicator data into the batch-stream integrated kafka;

[0007] Writing the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data;

[0008] Performing model training based on the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model;

[0009] Anomaly detection is performed on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model.

[0010] In order to achieve the above object, according to another aspect of the present invention, another batch-stream integrated indicator data anomaly detection method is provided, the method comprising:

[0011] Generate a data summary instance according to the preset data screening conditions and data processing method, wherein the data summary instance is used to screen the indicator data output by the data source Kafka according to the data screening conditions, process the screened indicator data according to the data processing method to obtain the processed indicator data, and input the processed indicator data into the batch-stream integrated Kafka;

[0012] Generate a big data processing instance corresponding to the data aggregation instance, wherein the big data processing instance is used to: write the processed indicator data output by the batch-stream integrated Kafka into the data inventory table corresponding to the processed indicator data; perform model training according to the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model; perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model.

[0013] In order to achieve the above object, according to another aspect of the present invention, a batch-stream integrated indicator data anomaly detection device is provided, the device comprising:

[0014] The indicator data processing unit is used to filter the indicator data output by the data source kafka according to the preset data filtering conditions, and process the filtered indicator data according to the preset data processing method to obtain the processed indicator data;

[0015] A sending unit, used for inputting the processed indicator data into the batch-stream integrated kafka;

[0016] A data storage unit, used to write the processed indicator data output by the batch-stream integrated Kafka into a data inventory table corresponding to the processed indicator data;

[0017] An indicator data prediction model training unit is used to perform model training according to the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model;

[0018] An anomaly detection unit is used to perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model.

[0019] In order to achieve the above object, according to another aspect of the present invention, another batch-stream integrated indicator data anomaly detection device is provided, the device comprising:

[0020] A data summary instance generation unit is used to generate a data summary instance according to a preset data screening condition and a data processing method, wherein the data summary instance is used to screen the indicator data output by the data source Kafka according to the data screening condition, process the screened indicator data according to the data processing method to obtain the processed indicator data, and input the processed indicator data into the batch-stream integrated Kafka;

[0021] A big data processing instance generation unit is used to generate a big data processing instance corresponding to the data aggregation instance, wherein the big data processing instance is used to: write the processed indicator data output by the batch-stream integrated Kafka into the data inventory table corresponding to the processed indicator data; perform model training according to the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model; perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model.

[0022] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned batch-stream integrated indicator data anomaly detection method when executing the computer program.

[0023] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed in a computer processor, the steps in the above-mentioned batch-stream integrated indicator data anomaly detection method are implemented.

[0024] The beneficial effects of the present invention are as follows: the embodiment of the present invention sets up a batch-stream integrated Kafka, writes the processed indicator data output by the batch-stream integrated Kafka into the corresponding data stock table, performs batch processing on the indicator data in the data stock table to train an indicator data prediction model, and then performs stream processing anomaly detection on the indicator data output by the batch-stream integrated Kafka according to the indicator data prediction model, thereby realizing the integration of batch processing tasks and stream processing tasks for indicator data anomaly detection, simplifying the development and operation and maintenance of indicator data anomaly detection, and helping to improve the efficiency and detection effect of indicator data anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 This is a first flow chart of a method for detecting abnormality of batch and stream integrated indicator data according to an embodiment of the present invention;

[0027] Figure 2 This is a second flow chart of the batch-stream integrated indicator data anomaly detection method according to an embodiment of the present invention;

[0028] Figure 3 It is an architecture diagram of batch-stream integrated indicator data anomaly detection according to an embodiment of the present invention;

[0029] Figure 4 This is a first structural block diagram of a batch-stream integrated indicator data anomaly detection device according to an embodiment of the present invention;

[0030] Figure 5 This is a second structural block diagram of the batch-stream integrated indicator data anomaly detection device according to an embodiment of the present invention;

[0031] Figure 6 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0034] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0035] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] The present invention provides an efficient and novel batch-stream integrated indicator data anomaly detection method to deal with the situation of "difficult to maintain and develop two independent stream data processing and batch data processing architectures". The present invention sets up a batch-stream integrated processing architecture, making most operation and maintenance indicator big data processing scenarios easy to develop and maintain. Regularly executed batch processing tasks provide guarantees for the accuracy of the model and the accuracy of historical anomaly discovery, and stream processing tasks provide guarantees for the timeliness of discovery of real-time anomalies. The present invention overcomes the shortcomings of splitting stream data processing and batch data processing into two different independent architectures, effectively reducing the pressure on operation and maintenance personnel and developers, and improving production and operation efficiency.

[0037] Figure 3 is an architecture diagram of the batch-stream integrated indicator data anomaly detection according to an embodiment of the present invention. Figure 3 As shown, the batch-stream integrated indicator data anomaly detection architecture of the present invention includes: an indicator data collection tool, a data source kafka, an indicator data processing device, a batch-stream integrated kafka and a TiDB database.

[0038] The indicator data collection tool can use tools such as Prometheus. The indicator data collection tool will uniformly clean the collected indicator data, form indicator data in a unified format, and then push it into the data source Kafka.

[0039] The indicator data processing device is used to perform batch processing and stream processing tasks to detect abnormal indicator data. Specifically, the indicator data processing device is used to filter and process the data output by the data source kafka, and send the processed indicator data to the batch-stream integrated kafka. Then, the indicator data processing device stores the data output by the batch-stream integrated kafka in the TiDB database, trains the indicator data prediction model based on the batch indicator data stored in the TiDB database, and performs stream processing on the indicator data output by the batch-stream integrated kafka according to the indicator data prediction model. Abnormal detection.

[0040] The data source Kafka and the batch-stream integrated Kafka are two Kafka message queues.

[0041] Figure 1 is a first flow chart of the batch-stream integrated indicator data anomaly detection method according to an embodiment of the present invention. Figure 1 As shown, in one embodiment of the present invention, the batch-stream integrated indicator data anomaly detection method of the present invention includes steps S101 to S105. The implementation subject of steps S101 to S105 is Figure 3 Indicator data processing device in.

[0042] Step S101, filter the indicator data output by the data source kafka according to the preset data filtering condition, and process the filtered indicator data according to the preset data processing method to obtain the processed indicator data.

[0043] like Figure 3 As shown, in one embodiment of the present invention, the indicator data processing device generates a data aggregation instance according to the data screening conditions and data processing methods set by the user. The data aggregation instance filters and processes the indicator data output by the data source kafka, and then sends it to the batch-stream integrated kafka.

[0044] In one embodiment of the present invention, the data screening condition includes: an identification field. In this step, the indicator data output by the data source kafka is screened according to the data screening condition, specifically, all indicator data containing the identification field are screened out.

[0045] In one embodiment of the present invention, the data processing method includes: filtering fields, wherein the filtering fields represent fields in the indicator data that need to be filtered out. This step processes the filtered indicator data according to the data processing method, specifically, filtering out the filtering fields in the filtered indicator data. In another embodiment of the present invention, the data processing method also includes: missing value filling method, wherein the missing value filling method is used to indicate the method used to fill in the missing fields in the indicator data.

[0046] In a specific embodiment of the present invention, the indicator data processing device creates and generates a data aggregation instance according to the topic of the data source Kafka, the broker IP of the data source Kafka, the identification field, the filter field, the missing value filling method and other information set by the user, and the aggregation instance name is consistent with the identification field. Starting the data aggregation instance will automatically push the data output by the data source Kafka into the batch-stream integrated Kafka according to the set fields and data filling method. The indicator data processing device can also control the start and stop of the data aggregation instance, as well as delete the data aggregation instance.

[0047] Step S102: input the processed indicator data into the batch-stream integrated Kafka.

[0048] In the embodiment of the present invention, after creating a data aggregation instance, starting the data aggregation instance will automatically filter and process the data output by the data source Kafka according to the set settings and push the data into the batch-stream integrated Kafka.

[0049] Step S103, writing the processed indicator data output by the batch-stream integrated Kafka into the data inventory table corresponding to the processed indicator data.

[0050] like Figure 3 As shown, in one embodiment of the present invention, the indicator data processing device performs batch processing and stream processing by creating a big data processing instance to realize abnormality inspection of the indicator data. In the embodiment of the present invention, the indicator data processing device needs to create a big data processing instance according to the data aggregation instance selected by the user, the indicator data prediction algorithm selected by the user, and the parameters of the set algorithm.

[0051] In one embodiment of the present invention, the big data processing instance includes three subtasks: Spark Streaming task, Spark task and Spark Streaming real-time stream processing task. Among them, the Spark Streaming task is used to write the processed index data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed index data.

[0052] In one embodiment of the present invention, the SparkStreaming task will also determine whether there is a data stock table corresponding to the processed indicator data. If there is no data stock table corresponding to the processed indicator data, the present invention generates a table name according to the identification field, creates a data stock table corresponding to the processed indicator data, and uses other fields in the processed indicator data except the filter field as table fields of the data stock table.

[0053] In one embodiment of the present invention, in this step, the field values ​​of other fields in the processed indicator data except the filter field may be written into the table fields of the data stock table.

[0054] In one embodiment of the present invention, after the SparkStreaming task is started, a data stock table is created in the TiDB database with the identification field as the table name. The fields of the data stock table are other fields corresponding to the identification field in the indicator data output from the batch-stream integrated kafka (i.e. other fields in the indicator data except the filter field); secondly, the SparkStreaming task determines whether the data stock table exists. If the table exists, the SparkStreaming task continues to execute, and the value of the field corresponding to the identification field is written into the data stock table of the tidb database. The SparkStreaming task is a resident task, and during its operation, the data output by the batch-stream integrated kafka is continuously written into the data stock table.

[0055] Step S104: Model training is performed according to the indicator data in the data stock table and a preset indicator data prediction algorithm to train an indicator data prediction model.

[0056] In one embodiment of the present invention, the Spark task in the big data processing instance is used to perform model training based on the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model.

[0057] The main function of the Spark task is to regularly use the data in the data stock table related to the big data processing instance as training data, implement the training of the algorithm model related to the big data processing instance, and generate a dynamic baseline and write it into the prediction table in the TiDB database. Specifically, after the Spark task is started, the data in the corresponding tidb database data stock table within the set time period is used as training data, and the indicator data prediction algorithm selected by the user when the new big data instance is created and stored in hdfs is loaded. The parameters of the algorithm set when the new big data instance is created are used as hyperparameters to perform model training and obtain the indicator data prediction model. The indicator data prediction model is used to predict the dynamic baseline and write the dynamic baseline into the prediction table in the tidb database.

[0058] In one embodiment of the present invention, the dynamic baseline includes three curves, an indicator data prediction value curve, an indicator maximum normal value curve, and an indicator minimum normal value curve. The indicator data prediction value curve includes the indicator data prediction value at each time point, the indicator maximum normal value curve includes the indicator maximum normal value at each time point, and the indicator minimum normal value curve includes the indicator minimum normal value at each time point. The indicator maximum normal value curve and the indicator minimum normal value curve constitute the indicator normal value range at each time point.

[0059] In the embodiment of the present invention, the predicted value of the indicator data refers to the predicted value of other fields in the indicator data except the filter field. The normal value range of the indicator refers to the normal value range of the field value of other fields in the indicator data except the filter field.

[0060] Step S105, performing anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model.

[0061] In an embodiment of the present invention, the Spark Streaming real-time stream processing task in the big data processing instance is used to perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model. The main function of the Spark Streaming real-time stream processing task is to realize real-time anomaly detection of the indicator data output by the batch-stream integrated Kafka. The specific process is: after the Spark Streaming stream processing task is started, the identification field output in the batch-stream integrated Kafka is used as an identifier, and the field values ​​of other fields except the filter field in the indicator data output by the batch-stream integrated Kafka are compared with the normal value range of the field value at the corresponding time point in the corresponding TiDB prediction table. If it is outside the normal value range, it is marked as an abnormal value. If it is within the normal value range, it is marked as a normal value, and the marked abnormal data is written into the abnormal table in the TiDB database.

[0062] In one embodiment of the present invention, this step may include the following steps when implemented:

[0063] Obtaining a dynamic baseline output by the indicator data prediction model, wherein the dynamic baseline includes indicator data prediction values ​​at multiple time points and an indicator normal value range;

[0064] The processed indicator data currently output by the batch-stream integrated kafka is compared with the normal value range of the indicator corresponding to the current time point in the dynamic baseline. If the processed indicator data currently output is not within the normal value range of the indicator corresponding to the current time point, it is determined that the processed indicator data currently output is abnormal.

[0065] The embodiment of the present invention writes the processed indicator data output by the batch-stream integrated Kafka into the corresponding data stock table, and performs batch processing on the indicator data in the data stock table to train an indicator data prediction model, and then performs stream processing anomaly detection on the indicator data output by the batch-stream integrated Kafka according to the indicator data prediction model, thereby realizing the integration of batch processing tasks and stream processing tasks for indicator data anomaly detection, simplifying the development and operation and maintenance of indicator data anomaly detection, and helping to improve the efficiency and detection effect of indicator data anomaly detection.

[0066] In one embodiment of the present invention, the specific process of generating and managing the above-mentioned data aggregation instance and big data processing instance includes the following steps S301 to S307.

[0067] Step S301, create a "data aggregation instance". When creating a data aggregation instance, you need to enter the topic of the data source Kafka, broker IP, identification field, filter field, missing value filling method and other information. After the creation is completed, a "data aggregation instance" is generated. The data aggregation instance name is consistent with the identification field name.

[0068] Step S302, start the "Data Aggregation Instance", click Start to start the corresponding data aggregation instance task. The backend will automatically push the data output by the data source Kafka into the batch-stream integrated Kafka according to the set fields and data filling methods.

[0069] Step S303: When there is no need to aggregate the data output from the Kafka of a certain profession, click the stop button to stop data aggregation.

[0070] Step S304, when it is necessary to re-screen the fields for processing, the data summary instance can be deleted and a new data summary instance that meets the requirements can be created.

[0071] Step S304: After the data aggregation task is started, it means that the data collected by each profession has been pushed into the batch-stream integrated Kafka according to the preset format standard requirements. At this time, a new big data processing instance can be created. When creating a new big data processing instance, you need to select a data aggregation instance, that is, to process the data aggregated by the data aggregation instance in the batch-stream integrated framework. After selecting the data aggregation instance, you need to select an algorithm module. After the algorithm module is selected, the algorithm hyperparameters that need to be exposed to the user need to be used. If the user does not fill in the default parameters, the default parameters are used.

[0072] Step S305, click the start button to start the selected batch and stream integrated big data processing instance to complete the batch and stream processing of the aggregated data.

[0073] Step S306: If you do not need to perform batch and stream processing on the data summarized by a data summary instance temporarily, you can select the batch and stream integrated big data processing instance and click Stop. If a data summary instance is deleted and you no longer need to perform batch and stream processing on the summarized data, select the batch and stream integrated big data processing instance and click Delete.

[0074] Step S307: When it is necessary to reselect the algorithm or readjust the parameters in the batch-stream integrated big data processing instance, the batch-stream integrated big data processing instance can be selected and the parameter configuration can be clicked to set the changed parameters. After starting the batch-stream integrated big data processing instance, the batch-stream integrated big data processing device will perform big data batch-stream integrated processing.

[0075] In one embodiment of the present invention, the present invention can also visualize the currently executed tasks and the indicator anomaly detection results. The specific visualization display is divided into two modules: task visualization and data visualization.

[0076] The task visualization module displays the overall running status of the data aggregation instance and the big data processing instance. The data aggregation instance corresponds to the batch-stream integrated big data processing instance one by one. When the instance is running normally, it is displayed in green, when the instance is suspended, it is displayed in gray, and when the instance is permanently deleted, it is not displayed. Click on the instance to display the data processing information details corresponding to the instance.

[0077] The data visualization module requires you to select a big data processing instance and time first. After selection, the real-time data processed by the big data processing instance and real-time abnormal data will be displayed. When the data processed in the big data processing instance has multiple performance dimensions and indicators, click the indicator to display the abnormal data and real-time data of each indicator of the performance indicator.

[0078] Based on the same inventive concept, the embodiment of the present invention also provides another batch-stream integrated indicator data anomaly detection method, such as Figure 2 As described in the embodiments. Figure 2 is a second flow chart of the batch-stream integrated indicator data anomaly detection method according to an embodiment of the present invention. Figure 2 As shown, in one embodiment of the present invention, the batch-stream integrated indicator data anomaly detection method of the present invention includes step S201 and step S202. The implementation subject of step S201 and step S202 can be Figure 3 Indicator data processing device in.

[0079] Step S201, generate a data summary instance according to preset data screening conditions and data processing methods, wherein the data summary instance is used to filter the indicator data output by the data source kafka according to the data screening conditions, process the filtered indicator data according to the data processing method to obtain the processed indicator data, and input the processed indicator data into the batch-stream integrated kafka.

[0080] In a specific embodiment of the present invention, the indicator data processing device creates and generates a data aggregation instance according to the topic of the data source Kafka, the broker IP of the data source Kafka, the identification field, the filter field, the missing value filling method and other information set by the user, and the aggregation instance name is consistent with the identification field. Starting the data aggregation instance will automatically push the data output by the data source Kafka into the batch-stream integrated Kafka according to the set fields and data filling method. The indicator data processing device can also control the start and stop of the data aggregation instance, as well as delete the data aggregation instance.

[0081] Step S202, generating a big data processing instance corresponding to the data aggregation instance, wherein the big data processing instance is used to: write the processed indicator data output by the batch-stream integrated Kafka into the data inventory table corresponding to the processed indicator data; perform model training according to the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model; perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model.

[0082] like Figure 3 As shown, in one embodiment of the present invention, the indicator data processing device performs batch processing and stream processing by creating a big data processing instance to realize abnormality inspection of the indicator data. In the embodiment of the present invention, the indicator data processing device needs to create a big data processing instance according to the data aggregation instance selected by the user, the indicator data prediction algorithm selected by the user, and the parameters of the set algorithm.

[0083] In one embodiment of the present invention, the big data processing instance includes three subtasks: Spark Streaming task, Spark task and Spark Streaming real-time stream processing task. The Spark Streaming task is used to write the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data. The main function of the Spark task is to regularly use the data in the data inventory table related to the big data processing instance as training data, implement the training of the algorithm model related to the big data processing instance, and generate a dynamic baseline to write into the prediction table in the TiDB database. The Spark Streaming real-time stream processing task is used to perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model.

[0084] because Figure 2 The principle and problem solving method of batch-stream integrated indicator data anomaly detection Figure 1 The same, so Figure 2 For an example of the batch-stream integrated indicator data anomaly detection method, see Figure 1 The embodiments of the method will not be described in detail.

[0085] In one embodiment of the present invention, the data screening condition includes: an identification field; and the data processing method includes: a filtering field.

[0086] In one embodiment of the present invention, the data aggregation instance is also used to generate a table name based on the identification field when there is no data stock table corresponding to the processed indicator data, create a data stock table corresponding to the processed indicator data, and use other fields in the processed indicator data except the filter field as table fields of the data stock table.

[0087] It can be seen from the above technical solutions that the efficient and novel batch-stream integrated indicator data anomaly detection method proposed by the present invention has great application value in improving the big data processing capabilities of IT industry operation and maintenance. The present invention has at least the following beneficial effects and advantages:

[0088] 1. The present invention bases all data consumption on Kafka, forms a unified data export, and finally forms a standardized data processing method to improve data processing efficiency.

[0089] 2. The present invention can display the execution status of the acquired task, and provide effective data and judgment basis for operation and maintenance personnel to monitor the task execution process.

[0090] 3. The present invention simplifies the operation and maintenance big data processing framework, forms a unified and standardized operation and maintenance big data processing framework, realizes the vision of integrated batch and stream processing of operation and maintenance big data, and reduces the operation and maintenance pressure of operation and maintenance personnel.

[0091] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0092] Based on the same inventive concept, the embodiment of the present invention also provides a batch-stream integrated indicator data anomaly detection device, which can be used to implement the above Figure 1 The batch-stream integrated indicator data anomaly detection method described in the embodiment. Since the principle of solving the problem by the batch-stream integrated indicator data anomaly detection device is similar to that of the batch-stream integrated indicator data anomaly detection method, the embodiment of the batch-stream integrated indicator data anomaly detection device can refer to the embodiment of the batch-stream integrated indicator data anomaly detection method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0093] Figure 4 is a first structural block diagram of a batch-stream integrated indicator data anomaly detection device according to an embodiment of the present invention, such as Figure 4 As shown, in one embodiment of the present invention, the batch-stream integrated indicator data anomaly detection device of the present invention includes:

[0094] The indicator data processing unit 1 is used to filter the indicator data output by the data source kafka according to the preset data filtering conditions, and process the filtered indicator data according to the preset data processing method to obtain the processed indicator data;

[0095] Sending unit 2, used to input the processed indicator data into batch and stream integrated kafka;

[0096] Data storage unit 3, used to write the processed indicator data output by the batch-stream integrated Kafka into a data inventory table corresponding to the processed indicator data;

[0097] The indicator data prediction model training unit 4 is used to perform model training according to the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model;

[0098] The anomaly detection unit 5 is used to perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model.

[0099] In one embodiment of the present invention, the data screening condition includes: an identification field; and the data processing method includes: a filtering field.

[0100] In one embodiment of the present invention, the batch-stream integrated indicator data anomaly detection device of the present invention further includes:

[0101] A data stock table establishment unit is used to generate a table name according to the identification field if there is no data stock table corresponding to the processed indicator data, create a data stock table corresponding to the processed indicator data, and use other fields in the processed indicator data except the filter field as table fields of the data stock table.

[0102] In one embodiment of the present invention, the data storage unit 3 is specifically used to write the field values ​​of other fields in the processed indicator data except the filter field into the table fields of the data stock table.

[0103] In one embodiment of the present invention, the abnormality detection unit 5 includes:

[0104] A dynamic baseline acquisition module, used to acquire the dynamic baseline output by the indicator data prediction model, wherein the dynamic baseline includes the indicator data prediction values ​​at multiple time points and the indicator normal value range;

[0105] The abnormality determination module is used to compare the processed indicator data currently output by the batch-stream integrated Kafka with the normal value range of the indicator corresponding to the current time point in the dynamic baseline. If the processed indicator data currently output is not within the normal value range of the indicator corresponding to the current time point, it is determined that the processed indicator data currently output is abnormal.

[0106] Based on the same inventive concept, the embodiment of the present invention also provides another batch-stream integrated indicator data anomaly detection device, which can be used to implement the above Figure 2 The batch-stream integrated indicator data anomaly detection method described in the embodiment.

[0107] Figure 5 is a second structural block diagram of the batch-stream integrated indicator data anomaly detection device according to an embodiment of the present invention, such as Figure 5 As shown, in one embodiment of the present invention, the batch-stream integrated indicator data anomaly detection device of the present invention includes:

[0108] A data summary instance generating unit 6 is used to generate a data summary instance according to a preset data screening condition and a data processing method, wherein the data summary instance is used to screen the indicator data output by the data source Kafka according to the data screening condition, process the screened indicator data according to the data processing method to obtain the processed indicator data, and input the processed indicator data into the batch-stream integrated Kafka;

[0109] A big data processing instance generation unit 7 is used to generate a big data processing instance corresponding to the data aggregation instance, wherein the big data processing instance is used to: write the processed indicator data output by the batch-stream integrated Kafka into the data inventory table corresponding to the processed indicator data; perform model training according to the indicator data in the data inventory table and a preset indicator data prediction algorithm to train an indicator data prediction model; perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model.

[0110] In one embodiment of the present invention, the data aggregation instance 6 is also used to generate a table name according to the identification field when there is no data stock table corresponding to the processed indicator data, create a data stock table corresponding to the processed indicator data, and use other fields in the processed indicator data except the filter field as table fields of the data stock table.

[0111] In order to achieve the above object, according to another aspect of the present application, a computer device is also provided. Figure 6 As shown, the computer device includes a memory, a processor, a communication interface and a communication bus. A computer program that can be executed on the processor is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment method are implemented.

[0112] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0113] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiments.

[0114] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The one or more units are stored in the memory, and when executed by the processor, the method in the above embodiment is performed.

[0116] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiments, and will not be repeated here.

[0117] In order to achieve the above purpose, according to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps in the above batch-stream integrated indicator data anomaly detection method when executed in a computer processor. Those skilled in the art can understand that the implementation of all or part of the process in the above embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the process of the embodiment of each method as described above when executed. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0118] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A batch-stream integrated indicator data anomaly detection method, characterized in that: Applied to an indicator data processing device, the method comprises: The indicator data output by the data source kafka is filtered according to the preset data filtering conditions, and the filtered indicator data is processed according to the preset data processing method to obtain the processed indicator data; Input the processed indicator data into the batch-stream integrated kafka; The processed indicator data output by the batch-stream integrated kafka is written into the data inventory table corresponding to the processed indicator data through the Spark Streaming task; the indicator data processing device performs batch processing and stream processing by creating a big data processing instance to realize abnormality inspection of the indicator data; the indicator data processing device creates a big data processing instance according to the data aggregation instance selected by the user, the indicator data prediction algorithm selected by the user and the parameters of the set algorithm; the big data processing instance includes three subtasks: Spark Streaming task, Spark task and Spark Streaming real-time stream processing task; Through the Spark task, model training is performed according to the indicator data in the data stock table and the preset indicator data prediction algorithm to train the indicator data prediction model; after the Spark task is started, the data in the corresponding tidb database data stock table within the set time period is used as training data, and the indicator data prediction algorithm selected by the user when establishing the big data processing instance stored in hdfs is loaded, and the parameters of the algorithm set when establishing the big data processing instance are used as hyperparameters to perform model training to obtain the indicator data prediction model, which is used to predict the dynamic baseline, and the dynamic baseline is written into the prediction table in the tidb database; the dynamic baseline includes three curves, an indicator data prediction value curve, an indicator maximum normal value curve, and an indicator minimum normal value curve, the indicator data prediction value curve includes the indicator data prediction value at each time point, the indicator maximum normal value curve includes the indicator maximum normal value at each time point, the indicator minimum normal value curve includes the indicator minimum normal value at each time point, and the indicator maximum normal value curve and the indicator minimum normal value curve constitute the indicator normal value range at each time point; Perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model through the Spark Streaming real-time stream processing task; after the Spark Streaming real-time stream processing task is started, the identification field output in the batch-stream integrated Kafka is used as an identifier, and the field values ​​of other fields except the filter field in the indicator data output by the batch-stream integrated Kafka are compared with the normal value range of the field value at the corresponding time point in the prediction table in the corresponding tidb database. If it is outside the normal value range, it is marked as an abnormal value; if it is within the normal value range, it is marked as a normal value, and the marked abnormal data is written into the abnormal table in the TiDB database; The method of filtering the indicator data output by the data source Kafka according to the preset data filtering conditions, processing the filtered indicator data according to the preset data processing method, obtaining the processed indicator data, and inputting the processed indicator data into the batch-stream integrated Kafka specifically includes: Generate a data aggregation instance based on the topic of the data source Kafka, the broker IP of the data source Kafka, the identification field, the filter field, and the missing value filling method set by the user. The data aggregation instance name is consistent with the identification field. Start the data aggregation instance to push the indicator data output by the data source Kafka into the batch-stream integrated Kafka after filtering and processing according to the settings.

2. The batch-stream integrated indicator data anomaly detection method according to claim 1 is characterized in that: The data screening condition includes: an identification field; the data processing method includes: a filtering field.

3. The batch-stream integrated indicator data anomaly detection method according to claim 2 is characterized in that: Before writing the processed indicator data output by the batch-stream integrated Kafka into a data inventory table corresponding to the processed indicator data, the method further includes: If there is no data stock table corresponding to the processed indicator data, a table name is generated according to the identification field, a data stock table corresponding to the processed indicator data is created, and other fields in the processed indicator data except the filter field are used as table fields of the data stock table.

4. The batch-stream integrated indicator data anomaly detection method according to claim 2, characterized in that: The table fields of the data stock table are other fields in the processed indicator data except the filter field; The step of writing the processed indicator data output by the batch-stream integrated Kafka into a data inventory table corresponding to the processed indicator data includes: The field values ​​of other fields in the processed indicator data except the filter field are written into the table fields of the data stock table.

5. The batch-stream integrated indicator data anomaly detection method according to claim 1, characterized in that: The performing anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model includes: Obtaining a dynamic baseline output by the indicator data prediction model, wherein the dynamic baseline includes indicator data prediction values ​​at multiple time points and an indicator normal value range; The processed indicator data currently output by the batch-stream integrated kafka is compared with the normal value range of the indicator corresponding to the current time point in the dynamic baseline. If the processed indicator data currently output is not within the normal value range of the indicator corresponding to the current time point, it is determined that the processed indicator data currently output is abnormal.

6. A batch-stream integrated indicator data anomaly detection method, characterized in that: Applied to an indicator data processing device, the method comprises: Generate a data summary instance according to the preset data screening conditions and data processing method, wherein the data summary instance is used to screen the indicator data output by the data source Kafka according to the data screening conditions, process the screened indicator data according to the data processing method to obtain the processed indicator data, and input the processed indicator data into the batch-stream integrated Kafka; Generate a big data processing instance corresponding to the data aggregation instance, wherein the big data processing instance is used to: write the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data; perform model training according to the indicator data in the data inventory table and the preset indicator data prediction algorithm to train the indicator data prediction model; perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model; the indicator data processing device performs batch processing and stream processing by creating a big data processing instance to realize anomaly detection on the indicator data; the indicator data processing device creates a big data processing instance according to the data aggregation instance selected by the user, the indicator data prediction algorithm selected by the user and the parameters of the set algorithm; the big data processing instance includes three subtasks: Spark Streaming task, Spark task and Spark Streaming real-time stream processing task; the Spark Streaming task is used to write the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data; the Spark task is used to detect the abnormality of the processed indicator data according to the data inventory table The indicator data in the tidb database and the preset indicator data prediction algorithm are used for model training to train the indicator data prediction model; after the Spark task is started, the data in the corresponding tidb database data stock table within the set time period is used as training data, and the indicator data prediction algorithm selected by the user when establishing the big data processing instance stored in hdfs is loaded, and the parameters of the algorithm set when establishing the big data processing instance are used as hyperparameters to perform model training to obtain the indicator data prediction model, which is used to predict the dynamic baseline and write the dynamic baseline into the prediction table in the tidb database; the dynamic baseline includes three curves, an indicator data prediction value curve, an indicator maximum normal value curve, and an indicator minimum normal value curve, the indicator data prediction value curve includes the indicator data prediction value at each time point, the indicator maximum normal value curve includes the indicator maximum normal value at each time point, the indicator minimum normal value curve includes the indicator minimum normal value at each time point, and the indicator maximum normal value curve and the indicator minimum normal value curve constitute the indicator normal value range at each time point; the Spark Streaming real-time stream processing task is used to perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model;After the SparkStreaming real-time stream processing task is started, the identification field output in the batch-stream integrated Kafka is used as the identification field, and the field values ​​of other fields in the indicator data output by the batch-stream integrated Kafka except the filter field are compared with the normal value range of the field value at the corresponding time point in the prediction table in the corresponding TiDB database. If it is outside the normal value range, it is marked as an abnormal value. If it is within the normal value range, it is marked as a normal value, and the marked abnormal data is written to the abnormal table in the TiDB database; The data aggregation instance is generated according to the preset data screening conditions and data processing methods, specifically including: Generate a data aggregation instance based on the topic of the data source Kafka, the broker IP of the data source Kafka, the identification field, the filter field, and the missing value filling method set by the user. The data aggregation instance name is consistent with the identification field. Start the data aggregation instance to push the indicator data output by the data source Kafka into the batch-stream integrated Kafka after filtering and processing according to the settings.

7. The batch-stream integrated indicator data anomaly detection method according to claim 6, characterized in that: The data screening condition includes: an identification field; the data processing method includes: a filtering field.

8. The batch-stream integrated indicator data anomaly detection method according to claim 7, characterized in that: The data aggregation instance is also used to generate a table name according to the identification field when there is no data stock table corresponding to the processed indicator data, create a data stock table corresponding to the processed indicator data, and use other fields in the processed indicator data except the filter field as table fields of the data stock table.

9. A batch-stream integrated indicator data anomaly detection device, characterized in that: Applied to an indicator data processing device, the batch-stream integrated indicator data anomaly detection device comprises: The indicator data processing unit is used to filter the indicator data output by the data source kafka according to the preset data filtering conditions, and process the filtered indicator data according to the preset data processing method to obtain the processed indicator data; A sending unit, used for inputting the processed indicator data into the batch-stream integrated kafka; A data storage unit is used to write the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data through a SparkStreaming task; the indicator data processing device performs batch processing and stream processing by creating a big data processing instance to realize abnormality inspection of the indicator data; the indicator data processing device creates a big data processing instance according to the data aggregation instance selected by the user, the indicator data prediction algorithm selected by the user, and the parameters of the set algorithm; the big data processing instance includes three subtasks: SparkStreaming task, Spark task, and SparkStreaming real-time stream processing task; The indicator data prediction model training unit is used to perform model training according to the indicator data in the data stock table and the preset indicator data prediction algorithm through the Spark task, and train the indicator data prediction model; after the Spark task is started, the data in the corresponding tidb database data stock table within the set time period is used as the training data, and the indicator data prediction algorithm selected by the user when establishing the big data processing instance stored in hdfs is loaded, and the parameters of the algorithm set when establishing the big data processing instance are used as hyperparameters to perform model training to obtain the indicator data prediction model, the indicator data prediction model is used to predict the dynamic baseline, and the dynamic baseline is written into the prediction table in the tidb database; the dynamic baseline includes three curves, an indicator data prediction value curve, an indicator maximum normal value curve, and an indicator minimum normal value curve, the indicator data prediction value curve includes the indicator data prediction value at each time point, the indicator maximum normal value curve includes the indicator maximum normal value at each time point, the indicator minimum normal value curve includes the indicator minimum normal value at each time point, and the indicator maximum normal value curve and the indicator minimum normal value curve constitute the indicator normal value range at each time point; an anomaly detection unit, used to perform anomaly detection on the processed indicator data output by the batch-stream integrated Kafka according to the indicator data prediction value output by the indicator data prediction model through the Spark Streaming real-time stream processing task; after the Spark Streaming real-time stream processing task is started, the identification field output in the batch-stream integrated Kafka is used as an identifier, and the field values ​​of other fields except the filter field in the indicator data output by the batch-stream integrated Kafka are compared with the normal value range of the field value at the corresponding time point in the prediction table in the corresponding tidb database. If it is outside the normal value range, it is marked as an abnormal value, and if it is within the normal value range, it is marked as a normal value, and the marked abnormal data is written into the abnormal table in the TiDB database; The indicator data processing unit is specifically used to generate a data summary instance according to the topic of the data source Kafka, the broker IP of the data source Kafka, the identification field, the filter field, and the missing value filling method set by the user, and the data summary instance name is consistent with the identification field; The sending unit is specifically used to start the data aggregation instance to push the indicator data output by the data source Kafka into the batch-stream integrated Kafka after the set filtering and processing.

10. A batch-stream integrated indicator data anomaly detection device, characterized in that: Applied to an indicator data processing device, the batch-stream integrated indicator data anomaly detection device comprises: A data summary instance generation unit is used to generate a data summary instance according to a preset data screening condition and a data processing method, wherein the data summary instance is used to screen the indicator data output by the data source Kafka according to the data screening condition, process the screened indicator data according to the data processing method to obtain the processed indicator data, and input the processed indicator data into the batch-stream integrated Kafka; A big data processing instance generation unit is used to generate a big data processing instance corresponding to the data aggregation instance, wherein the big data processing instance is used to: write the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data; perform model training according to the indicator data in the data inventory table and the preset indicator data prediction algorithm to train the indicator data prediction model; perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model; the indicator data processing device performs batch processing and stream processing by creating a big data processing instance to realize anomaly detection of the indicator data; the indicator data processing device creates a big data processing instance according to the data aggregation instance selected by the user, the indicator data prediction algorithm selected by the user and the parameters of the set algorithm; the big data processing instance includes three subtasks: Spark Streaming task, Spark task and Spark Streaming real-time stream processing task; the Spark Streaming task is used to write the processed indicator data output by the batch-stream integrated kafka into the data inventory table corresponding to the processed indicator data; the Spark task is used to detect the abnormality of the processed indicator data according to the predicted value of the indicator data output by the indicator data prediction model; The indicator data in the data stock table and the preset indicator data prediction algorithm are used for model training to train the indicator data prediction model; after the Spark task is started, the data in the corresponding tidb database data stock table within the set time period is used as training data, and the indicator data prediction algorithm selected by the user when establishing the big data processing instance stored in hdfs is loaded, and the parameters of the algorithm set when establishing the big data processing instance are used as hyperparameters to perform model training to obtain the indicator data prediction model, which is used to predict the dynamic baseline and write the dynamic baseline into the prediction table in the tidb database; the dynamic baseline includes three curves, an indicator data prediction value curve, an indicator maximum normal value curve, and an indicator minimum normal value curve, the indicator data prediction value curve includes the indicator data prediction value at each time point, the indicator maximum normal value curve includes the indicator maximum normal value at each time point, the indicator minimum normal value curve includes the indicator minimum normal value at each time point, and the indicator maximum normal value curve and the indicator minimum normal value curve constitute the indicator normal value range at each time point; the SparkStreaming real-time stream processing task is used to perform anomaly detection on the processed indicator data output by the batch-stream integrated kafka according to the indicator data prediction value output by the indicator data prediction model;After the SparkStreaming real-time stream processing task is started, the identification field output in the batch-stream integrated Kafka is used as the identification field, and the field values ​​of other fields in the indicator data output by the batch-stream integrated Kafka except the filter field are compared with the normal value range of the field value at the corresponding time point in the prediction table in the corresponding TiDB database. If it is outside the normal value range, it is marked as an abnormal value. If it is within the normal value range, it is marked as a normal value, and the marked abnormal data is written to the abnormal table in the TiDB database; The data aggregation instance generation unit is specifically used to generate a data aggregation instance according to the topic of the data source Kafka, the broker IP of the data source Kafka, the identification field, the filter field, and the missing value filling method set by the user. The data aggregation instance name is consistent with the identification field. The data aggregation instance is started to push the indicator data output by the data source Kafka into the batch-stream integrated Kafka after filtering and processing according to the set settings.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed in a computer processor, the computer program implements the method according to any one of claims 1 to 8.

Images