A data processing method, device, equipment and storage medium

Abstract

The present application relates to a data processing method, device, equipment and storage medium, the method comprising: obtaining a set of data to be analyzed in a preset format; obtaining a set of rules; matching the set of data to be analyzed with the set of rules according to a mapping relationship to obtain a matching data set; the mapping relationship includes the identification of each data to be analyzed in the set of data to be analyzed and the identification of the rule corresponding to the identification of each data to be analyzed; the matching data set includes each data to be analyzed and the type of rule corresponding to each data to be analyzed; based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is an alarm type; and alarm prompt information is generated based on the alarm data. By performing reasonable analysis and processing on the log data through the above steps, the effectiveness of the log data processing process can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In enterprise-level business maintenance and operation scenarios, logs are playing an increasingly important role. Simple local storage of business logs makes it difficult to mine the real data value behind the logs. Therefore, enterprises collect and aggregate logs on all servers for centralized log management.

[0003] After storing logs on a centralized server, how to process them into indicators to guide maintenance and operations has become an increasingly urgent need for enterprises.

[0004] At present, traditional log statistics and analysis systems can no longer meet the core needs of enterprise users. How to efficiently process and use log data is an issue that needs to be addressed urgently. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, apparatus, device and storage medium, which can improve the effectiveness of the log data processing process by reasonably analyzing and processing the log data.

[0006] On the one hand, an embodiment of the present application provides a data processing method, including:

[0007] Obtain a set of data to be analyzed in a preset format;

[0008] Obtain a rule set; the rule set includes multiple rules and an identifier and a type of each rule in the multiple rules;

[0009] The data set to be analyzed is matched with the rule set according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the data set to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of the rule corresponding to each data to be analyzed;

[0010] Based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type;

[0011] Generate alarm prompt information based on alarm data.

[0012] Optionally, obtain the data to be analyzed in a preset format, including:

[0013] Get the log data corresponding to the indicator type from the message middleware;

[0014] Parse the log data to obtain the data to be analyzed in a preset format.

[0015] Optionally, get a set of rules, including:

[0016] Obtaining broadcast information from a preset storage area; the broadcast information includes a set of rules;

[0017] The preset storage area includes a Mysql database or an Elasticsearch database.

[0018] Optionally, the identifier of each data to be analyzed includes an application identifier and a data identifier;

[0019] According to the mapping relationship, the data set to be analyzed is matched with the rule set to obtain a matching data set, including:

[0020] For each data to be analyzed in the data set to be analyzed: according to the application identifier and data identifier of each data to be analyzed, determine the identifier of the rule corresponding to each data to be analyzed from the mapping relationship;

[0021] According to the identifier of the rule corresponding to each data to be analyzed, the rule corresponding to each data to be analyzed is obtained from the rule set, and the rule corresponding to each data to be analyzed is assembled with each data to be analyzed to obtain a matching data set.

[0022] Optionally, based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated to obtain alarm data, including:

[0023] For each data to be analyzed: if the type of the rule corresponding to the data to be analyzed is an alarm type, the data to be analyzed is determined as alarm data to be judged, and a set of alarm data to be judged is obtained;

[0024] According to the alarm type corresponding to each alarm data to be judged in the alarm data set to be judged, a first aggregation process is performed to obtain multiple subsets of alarm data to be judged; wherein each subset of alarm data to be judged in the multiple subsets of alarm data to be judged corresponds to a different alarm type; and each alarm data to be judged in each subset of alarm data to be judged corresponds to the same alarm type;

[0025] For each subset of alarm data to be judged: perform a second aggregation process to obtain the alarm data corresponding to the subset of alarm data to be judged.

[0026] Optionally, generate alarm prompt information based on the alarm data, including:

[0027] Obtain historical reference data;

[0028] Determine the rate of change based on alarm data and historical reference data;

[0029] If the change rate does not meet the preset range, a corresponding alarm prompt message is generated.

[0030] Optionally, based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated, further comprising:

[0031] For each data to be analyzed: if the type of the rule corresponding to the data to be analyzed is a statistical type, the data to be analyzed is determined as statistical data to be analyzed, and a set of statistical data to be analyzed is obtained;

[0032] Aggregate the statistical data set;

[0033] The aggregated statistics are sent to the preset storage area for storage.

[0034] On the other hand, an embodiment of the present application provides a data processing device, including:

[0035] A first acquisition module, used to acquire a set of data to be analyzed in a preset format;

[0036] A second acquisition module is used to acquire a rule set; the rule set includes multiple rules and an identifier and a type of each rule in the multiple rules;

[0037] The first determination module is used to match the data set to be analyzed with the rule set according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the data set to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of the rule corresponding to each data to be analyzed;

[0038] The second determination module is used to aggregate the matching data set based on the type of rule corresponding to each data to be analyzed to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type;

[0039] The generation module is used to generate alarm prompt information based on the alarm data.

[0040] On the other hand, an embodiment of the present application provides a device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor to execute the above-mentioned data processing method.

[0041] On the other hand, an embodiment of the present application provides a computer storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned data processing method.

[0042] The data processing method, device, equipment and storage medium provided by the embodiments of the present application have the following beneficial effects:

[0043] By obtaining a set of data to be analyzed in a preset format; obtaining a set of rules; the set of rules includes multiple rules and the identifier and type of each rule in the multiple rules; matching the set of data to be analyzed with the set of rules according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the set of data to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of rule corresponding to each data to be analyzed; based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type; and alarm prompt information is generated based on the alarm data. By performing reasonable analysis and processing on the log data through the above steps, the effectiveness of the log data processing process can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of a process for obtaining a set of data to be analyzed in a preset format provided by an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of a data processing framework provided in an embodiment of the present application;

[0048] Figure 4 This is a flowchart of how to obtain a matching data set provided by an embodiment of the present application;

[0049] Figure 5 This is a flowchart of how to obtain alarm data provided by an embodiment of the present application;

[0050] Figure 6 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0051] Figure 7 This is a schematic diagram of a process for generating alarm prompt information based on alarm data provided by an embodiment of the present application;

[0052] Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0053] Fig. 9 It is a hardware structure block diagram of a server of a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] The object of data processing in this application mainly refers to log data, which can include system log data, application log data and security log data. System operation and development personnel can use log data to understand server hardware and software information, check errors in the configuration process and the causes of errors, and take timely measures to correct errors; operation personnel can use log data to understand various business indicators, so as to better solve practical business problems.

[0057] In the embodiment of the present application, a set of data to be analyzed and a set of rules in a preset format are obtained by a server; the set of rules includes multiple rules and the identifier and type of each rule in the multiple rules; then, the server matches the set of data to be analyzed with the set of rules according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the set of data to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of rule corresponding to each data to be analyzed; then, the server aggregates the matching data set based on the type of rule corresponding to each data to be analyzed to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type; finally, the server generates alarm prompt information based on the alarm data. Optionally, the above-mentioned server may include an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0058] The following describes a specific embodiment of a data processing method of the present application. Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order shown in the embodiment or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 1 As shown, the method may include:

[0059] S101: Obtain a set of to-be-analyzed data in a preset format.

[0060] In the embodiment of the present application, the data set to be analyzed includes multiple log data, and the structure of each log data in the multiple log data is in a preset format.

[0061] In an optional implementation manner, the above step S101, as Figure 2 As shown, it may include:

[0062] S1011: Obtain log data corresponding to the indicator type from the message middleware;

[0063] S1012: Parse the log data to obtain data to be analyzed in a preset format.

[0064] Specifically, Figure 3 As shown, Figure 3 It is a schematic diagram of a data processing framework provided in an embodiment of the present application. The above-mentioned message middleware can be Kafka, which is a high-throughput distributed publish-subscribe message system that can process large amounts of data in real time to meet various demand scenarios; the present application pre-uses a centralized log management method to collect log data on all servers through filebeat, and then sends the log data to the corresponding type (topic) in Kafka; for example, for indicator data appearing in actual business scenarios, store it under the indicator type, and when the indicator data needs to be analyzed, the corresponding log data can be obtained according to the indicator type; the above-mentioned preset format can be a key-value format; the obtained log data is parsed to obtain data in key-value format. For example, the original log data to be analyzed is "2021-05-10 22:29:35.299 Hangzhou|a public service|1". After parsing, the key-value formatted data to be analyzed is {"City":"Hangzhou","Business Line":"a Public Service","Order Volume":"100"}; in this way, by parsing the multiple original log data obtained, a set of data to be analyzed based on the key-value format is obtained.

[0065] S103: Acquire a rule set; the rule set includes multiple rules and an identifier and a type of each rule in the multiple rules.

[0066] In the embodiment of the present application, the rule set refers to the rule flow data. Figure 3 As shown, this application processes streaming data based on the Flink framework, that is, obtains regular stream data from an external system, and then matches and assembles multiple log data with the regular stream data to achieve the combination of regular stream and data stream. The specific processing logic can be defined in processfunction.

[0067] In an optional implementation manner, the above step S103 may include:

[0068] S1031: Acquire broadcast information from a preset storage area; the broadcast information includes a rule set;

[0069] The preset storage area includes a Mysql database or an Elasticsearch database.

[0070] Specifically, Figure 3As shown, a data source is predefined as the source of rule flow data. For example, MySQL database or Elasticsearch data can be used as the data source. Rule flow data is broadcast regularly, and a thread is set to regularly obtain rule sets. In this way, the rules can be updated in quasi-real time.

[0071] S105: Match the data set to be analyzed with the rule set according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the data set to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of the rule corresponding to each data to be analyzed.

[0072] In an embodiment of the present application, a mapping relationship is pre-configured, and the mapping relationship includes the identifier of each data to be analyzed in the data set to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed. In this way, according to the identifier of each data to be analyzed in the data set to be analyzed, the identifier of the rule corresponding to the identifier of each data to be analyzed can be determined according to the mapping relationship, and then the rule corresponding to the identifier of the rule is assembled with the data to be analyzed to form new matching data and obtain a matching data set; since each rule carries a type field, the matching data set can include each data to be analyzed and the type of the rule corresponding to each data to be analyzed.

[0073] In an optional implementation, each identifier of the data to be analyzed includes an application identifier and a data identifier; wherein the application identifier represents the name of the business corresponding to the data to be analyzed, and the data identifier represents the number of fields corresponding to the data to be analyzed; then, the above step S105, such as Figure 4 As shown, it may include:

[0074] S1051: for each data to be analyzed in the data set to be analyzed: according to the application identifier and data identifier of each data to be analyzed, determine the identifier of the rule corresponding to each data to be analyzed from the mapping relationship;

[0075] S1052: According to the identifier of the rule corresponding to each data to be analyzed, the rule corresponding to each data to be analyzed is obtained from the rule set, and the rule corresponding to each data to be analyzed is assembled with each data to be analyzed to obtain a matching data set.

[0076] Specifically, the mapping relationship can be shown in the following table:

[0077]

[0078] Among them, application name A and application name B are application identifiers, Rule1, Rule2 and Rule3 represent the identifiers of different rules; the corresponding rules can be determined by the application identifier and data identifier carried by the data to be analyzed; for example, application name A can be dispatcher, and correspondingly, data identifier 1 can be dispatcher-ten-field, data identifier 2 can be dispatcher-nine-field, and data identifier 1 and data identifier 2 represent log data with different numbers of fields; in this mapping relationship, the data identifier and the rule identifier are a one-to-many relationship, which means that each log data identifier can correspond to multiple rules, and the multiple rules have different types, such as alarm type and statistical type. The same log data is matched and assembled with different types of rules to realize the subsequent calculation of different business logics. It should be noted that according to the actual application scenario, different applications can freely configure multiple statistics and alarm rules.

[0079] S107: Based on the type of rule corresponding to each data to be analyzed, aggregate the matching data set to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type.

[0080] S109: Generate warning prompt information based on the warning data.

[0081] In the present application embodiment, Figure 3 As shown in the figure, by using the window feature of the Flink framework, you can easily define a time window to count events within a certain period of time, and aggregate the matching data set according to the type of rule corresponding to each data to be analyzed to obtain alarm data. When the time reaches the threshold within the time window, an alarm prompt message is generated.

[0082] In an optional implementation manner, the above step S107, as Figure 5 As shown, it may include:

[0083] S1071: for each data to be analyzed: if the type of the rule corresponding to the data to be analyzed is an alarm type, determine the data to be analyzed as alarm data to be judged, and obtain a set of alarm data to be judged;

[0084] S1072: performing a first aggregation process according to the alarm type corresponding to each alarm data to be judged in the alarm data set to be judged, to obtain multiple subsets of alarm data to be judged; wherein each subset of alarm data to be judged in the multiple subsets of alarm data to be judged corresponds to a different alarm type; and each alarm data to be judged in each subset of alarm data to be judged corresponds to the same alarm type;

[0085] S1073: For each subset of alarm data to be determined: perform a second aggregation process to obtain alarm data corresponding to the subset of alarm data to be determined.

[0086] The above optional implementation is described below through a specific example. Based on the mapping relationship shown in the above table, it is assumed that the data set to be analyzed includes the following data in key-value format:

[0087] If the identifier of data 1 includes application name A and data identifier 1, after matching and assembling according to the above mapping relationship, two matching data can be obtained, namely, matching data 1: data 1+Rule 1 and matching data 2: data 1+Rule 2;

[0088] If the identifier of data 2 includes application name A and data identifier 2, after matching and assembling according to the above mapping relationship, a matching data is obtained, that is, matching data 3: data 2+Rule 3;

[0089] If the identifier of data 3 includes application name B and data identifier 1, after matching and assembling according to the above mapping relationship, a matching data is obtained, that is, matching data 4: data 3+Rule 1;

[0090] During assembly, a new data number can be assigned to the matching data for subsequent diversion; this prevents subsequent aggregation operations from being concentrated on one thread for processing too much data, causing performance bottlenecks;

[0091] If the types of Rule1 and Rule2 are alarm types, then matching data 1, matching data 2 and matching data 4 are all alarm data to be judged, and matching data 1, matching data 2 and matching data 4 constitute an alarm data set to be judged; since matching data 1 and matching data 4 are the same rule, their alarm types must be the same. Assuming that Rule2 is an alarm type different from Rule1, after the first aggregation processing, two subsets of alarm data to be judged are obtained, namely subset 1 {matching data 1, matching data 4} and subset 2 {matching data 2}; during the second set processing, all matching data in subset 1 are calculated according to the definition of Rule1 to obtain the alarm data corresponding to subset 1, and the matching data in subset 2 are calculated according to the definition of Rule2 to obtain the alarm data corresponding to subset 2.

[0092] In an optional implementation manner, the above step S107, as Figure 6 As shown, it may also include:

[0093] S1074: for each data to be analyzed: if the type of the rule corresponding to the data to be analyzed is a statistical type, the data to be analyzed is determined as statistical data to be analyzed, and a set of statistical data to be analyzed is obtained;

[0094] S1075: performing aggregation processing on the statistical data set;

[0095] S1076: Send the aggregated statistics to be counted to a preset storage area for storage.

[0096] Specifically, the above example will be used to explain. Figure 3 As shown, assuming that the type of Rule3 is a statistical type, matching data 3 is determined as the statistical data to be counted, and matching data 3 constitutes a set of statistical data to be counted; matching data 3 is calculated according to the definition of Rule3 to obtain the statistical data to be counted, and then the statistical data to be counted is sent to a preset storage area for storage. The preset storage area can be Influxdb. Influxdb is an open source distributed time series, time and indicator database, which is often used for monitoring data statistics, etc.

[0097] In an optional implementation manner, the above step S109, as Figure 7 As shown, it may include:

[0098] S1091: Obtain historical reference data;

[0099] S1092: Determine a change rate according to the alarm data and historical reference data;

[0100] S1093: If the change rate does not meet the preset range, generate corresponding alarm prompt information.

[0101] Specifically, each data to be analyzed carries timestamp information, and the alarm data is obtained by aggregation through a preset time window. In order to further determine whether the alarm data is abnormal, historical reference data is obtained, which can be the alarm data of the same time period in the past. Then the current alarm data is calculated with the alarm data of the same time period in the past to determine the change rate, and then compared with the preset range. If the preset range is not met, an alarm prompt message is generated to notify the user. For example, the alarm data {"city":"Hangzhou","business line":"a official business","order volume":"100"} is counted according to the city, business line, and 1 minute (this time dimension is configurable) dimension according to the alarm rules, and then the order volume increases or decreases by more than 50% compared with yesterday, then an alarm prompt message is generated based on the alarm data to notify the user to pay attention in time.

[0102] In an optional implementation manner, after step S109, the method may further include:

[0103] S111: Send the alarm prompt information to the message middleware to notify the user through the message middleware.

[0104] Specifically, Figure 3As shown, the message middleware is Kafka, and when generating alarm prompt information, a corresponding alarm level can be generated. Kafka can send alarm prompt information to different terminals or different terminal applications bound to the user according to the priority of the alarm level. For example, when the alarm level priority is high, the terminal user is notified by phone, and when the alarm level priority is low, the terminal user is notified by SMS notification or application message push.

[0105] In an optional implementation manner, after step S109, the method may further include:

[0106] S113: Customize the statistical dimensions and statistical fields of the alarm data and / or statistical data according to the corresponding console business, and store the statistical data in the database; thus, the amount of data that needs to be stored can be greatly reduced;

[0107] S115: Use grafana to configure the dashboard to display a continuous curve chart of the data in the database;

[0108] Specifically, Grafana can realize visual monitoring and analysis capabilities. You can configure indicator curve charts on Grafana as needed to observe indicator trends.

[0109] A data processing method provided in an embodiment of the present application uses Kafka to effectively smooth out peaks and fill valleys, thereby preventing the program from crashing due to inability to process enough data when traffic suddenly increases. The back pressure function of Flink also controls the inflow of data, thereby avoiding crashes caused by inability to process enough data when the amount of data is large. It can also issue alarms when abnormal indicators appear, and at the same time, the amount of data stored in the warehouse after aggregation is reduced, thereby reducing the utilization rate of machine resources.

[0110] The present application also provides a data processing device. Figure 8 is a structural diagram of a data processing device provided in an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0111] The first acquisition module 801 is used to acquire a set of data to be analyzed in a preset format;

[0112] The second acquisition module 802 is used to acquire a rule set; the rule set includes multiple rules and the identifier and type of each rule in the multiple rules;

[0113] The first determination module 803 is used to match the data set to be analyzed with the rule set according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the data set to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of the rule corresponding to each data to be analyzed;

[0114] The second determination module 804 is used to aggregate the matching data set based on the type of rule corresponding to each data to be analyzed to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type;

[0115] The generating module 805 is used to generate warning prompt information based on the warning data.

[0116] In an optional implementation, the first acquisition module 801 is specifically used to: acquire log data corresponding to the indicator type from the message middleware; and parse the log data to obtain data to be analyzed in a preset format.

[0117] In an optional implementation, the second acquisition module 802 is specifically used to: acquire broadcast information from a preset storage area; the broadcast information includes a rule set; wherein the preset storage area includes a Mysql database or an Elasticsearch database.

[0118] In an optional implementation manner, the identifier of each data to be analyzed includes an application identifier and a data identifier; the first determination module 803 is specifically used to: for each data to be analyzed in the data set to be analyzed: according to the application identifier and the data identifier of each data to be analyzed, determine the identifier of the rule corresponding to each data to be analyzed from the mapping relationship;

[0119] According to the identifier of the rule corresponding to each data to be analyzed, the rule corresponding to each data to be analyzed is obtained from the rule set, and the rule corresponding to each data to be analyzed is assembled with each data to be analyzed to obtain a matching data set.

[0120] In an optional implementation manner, the second determining module 804 is specifically configured to:

[0121] For each data to be analyzed: if the type of the rule corresponding to the data to be analyzed is an alarm type, the data to be analyzed is determined as alarm data to be judged, and a set of alarm data to be judged is obtained;

[0122] According to the alarm type corresponding to each alarm data to be judged in the alarm data set to be judged, a first aggregation process is performed to obtain multiple subsets of alarm data to be judged; wherein each subset of alarm data to be judged in the multiple subsets of alarm data to be judged corresponds to a different alarm type; and each alarm data to be judged in each subset of alarm data to be judged corresponds to the same alarm type;

[0123] For each subset of alarm data to be judged: perform a second aggregation process to obtain the alarm data corresponding to the subset of alarm data to be judged.

[0124] In an optional implementation, the generation module 805 is specifically used to: obtain historical reference data; determine the change rate according to the alarm data and the historical reference data; if the change rate does not meet the preset range, generate corresponding alarm prompt information.

[0125] In an optional implementation, the second determination module 804 is further specifically used for: for each data to be analyzed: if the type of rule corresponding to the data to be analyzed is a statistical type, determining the data to be analyzed as statistical data to be analyzed, and obtaining a set of statistical data to be analyzed; performing aggregation processing on the set of statistical data; and sending the aggregated statistical data to be analyzed to a preset storage area for storage.

[0126] The device and method embodiments in the embodiments of this application are based on the same application concept.

[0127] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal, a server or a similar computing device. Taking running on a server as an example, Fig. 9 1 is a hardware structure diagram of a server of a data processing method provided in an embodiment of the present application. Fig. 9 As shown, the server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor NCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the server 900. The server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or one or more operating systems 921, such as Windows, Mac OS, Unix, Linux, FreeBSD, etc.

[0128] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0129] It can be understood by those skilled in the art that Fig. 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations shown.

[0130] An embodiment of the present application also provides a storage medium, which can be set in a server to store at least one instruction, at least one program, a code set or an instruction set related to a data processing method in an implementation method embodiment. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the above-mentioned data processing method.

[0131] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0132] It can be seen from the embodiments of the data processing method, device, equipment and storage medium provided by the present application that in the present application, a data set to be analyzed in a preset format is obtained; a rule set is obtained; the rule set includes multiple rules and the identifier and type of each rule in the multiple rules; the data set to be analyzed is matched with the rule set according to the mapping relationship to obtain a matching data set; the mapping relationship includes the identifier of each data to be analyzed in the data set to be analyzed and the identifier of the rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and the type of rule corresponding to each data to be analyzed; based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is an alarm type; and an alarm prompt message is generated based on the alarm data. By performing reasonable analysis and processing of log data through the above steps, the effectiveness of the log data processing process can be improved.

[0133] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0136] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: Obtain a set of data to be analyzed in a preset format; Obtain a rule set; the rule set includes multiple rules and an identifier and a type of each rule in the multiple rules; The set of data to be analyzed is matched with the set of rules according to a mapping relationship to obtain a matching data set; the mapping relationship includes an identifier of each data to be analyzed in the set of data to be analyzed and an identifier of a rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and a type of rule corresponding to each data to be analyzed; Based on the type of rule corresponding to each data to be analyzed, the matching data set is aggregated to obtain alarm data; The type of rule corresponding to each data to be analyzed in the alarm data is the alarm type; Generate warning prompt information based on the warning data; The step of matching the data set to be analyzed with the rule set according to the mapping relationship to obtain a matching data set includes: For each data to be analyzed in the set of data to be analyzed: determining an identifier of a rule corresponding to each data to be analyzed from the mapping relationship according to an application identifier and a data identifier of each data to be analyzed; According to the identifier of the rule corresponding to each data to be analyzed, the rule corresponding to each data to be analyzed is obtained from the rule set, and the rule corresponding to each data to be analyzed is assembled with each data to be analyzed to obtain the matching data set.

2. The method according to claim 1, characterized in that The step of obtaining the data to be analyzed in a preset format includes: Get the log data corresponding to the indicator type from the message middleware; The log data is parsed to obtain the data to be analyzed in the preset format.

3. The method according to claim 1, characterized in that The acquisition rule set includes: Acquire broadcast information from a preset storage area; the broadcast information includes the rule set; Wherein, the preset storage area includes a Mysql database or an Elasticsearch database.

4. The method according to claim 1, characterized in that: The identifier of each data to be analyzed includes an application identifier and a data identifier.

5. The method according to claim 1, characterized in that The step of aggregating the matching data set based on the type of rule corresponding to each of the data to be analyzed to obtain the alarm data includes: For each of the data to be analyzed: if the type of the rule corresponding to the data to be analyzed is an alarm type, the data to be analyzed is determined as alarm data to be judged, and a set of alarm data to be judged is obtained; According to the alarm type corresponding to each alarm data to be judged in the alarm data set to be judged, a first aggregation process is performed to obtain multiple subsets of alarm data to be judged; wherein each subset of alarm data to be judged in the multiple subsets of alarm data to be judged corresponds to a different alarm type; and each alarm data to be judged in each subset of alarm data to be judged corresponds to the same alarm type; For each subset of alarm data to be determined: performing a second aggregation process to obtain alarm data corresponding to the subset of alarm data to be determined.

6. The method according to claim 5, characterized in that The generating of the warning prompt information based on the warning data includes: Obtain historical reference data; determining a change rate according to the alarm data and the historical reference data; If the change rate does not meet the preset range, a corresponding alarm prompt message is generated.

7. The method according to claim 1, characterized in that The aggregating the matching data set based on the type of rule corresponding to each of the data to be analyzed further includes: For each of the data to be analyzed: if the type of the rule corresponding to the data to be analyzed is a statistical type, the data to be analyzed is determined as statistical data to be analyzed, and a set of statistical data to be analyzed is obtained; Performing aggregation processing on the set of statistical data to be counted; The aggregated statistics are sent to the preset storage area for storage.

8. A data processing device, characterized in that: include: A first acquisition module, used to acquire a set of data to be analyzed in a preset format; A second acquisition module is used to acquire a rule set; The rule set includes a plurality of rules and an identifier and a type of each rule in the plurality of rules; A first determination module is used to match the data set to be analyzed with the rule set according to a mapping relationship to obtain a matching data set; the mapping relationship includes an identifier of each data to be analyzed in the data set to be analyzed and an identifier of a rule corresponding to the identifier of each data to be analyzed; the matching data set includes each data to be analyzed and a type of rule corresponding to each data to be analyzed; The step of matching the set of data to be analyzed with the set of rules according to the mapping relationship to obtain a set of matching data includes: for each data to be analyzed in the set of data to be analyzed: determining the identifier of the rule corresponding to each data to be analyzed from the mapping relationship according to the application identifier and the data identifier of each data to be analyzed; obtaining the rule corresponding to each data to be analyzed from the rule set according to the identifier of the rule corresponding to each data to be analyzed, and assembling the rule corresponding to each data to be analyzed with each data to be analyzed to obtain the set of matching data; A second determination module is used to aggregate the matching data set based on the type of rule corresponding to each data to be analyzed to obtain alarm data; the type of rule corresponding to each data to be analyzed in the alarm data is the alarm type; A generating module is used to generate warning prompt information based on the warning data.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executes the data processing method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 7.

Images