Service abnormality alarm method and its device, equipment, medium, and product

Abstract

The present application relates to a service abnormality alarm method and its device, equipment, medium, and product, the method comprising: starting a feature detector in response to an online service startup event, for monitoring local log records generated by the online service; intercepting the local log records of the online service by the feature detector, and generating the category features to which the abnormal events represented by the local log records belong; the feature detector periodically sends a reporting event carrying the category features to the service controller, triggering the service controller to statistically update the total amount of reporting records corresponding to the category features according to the number of occurrences of the category features within the corresponding timing period; the service controller monitors whether the total amount of reporting records of each category feature exceeds a preset threshold, and when it exceeds the preset threshold, generates an alarm message corresponding to the category feature. The present application can balance the processing timeliness and data scale required for online service abnormality alarms, and timely discover abnormal events.

Description

Technical Field

[0001] The present application relates to network security technology, and in particular to a service abnormality alarm method and its device, equipment, medium, and product. Background Art

[0002] Large Internet platforms usually deploy online services with the help of microservice architecture. Massive online services enrich the functions of the platform in all aspects and ensure the stable operation of the platform. For the needs of platform maintenance, online services will pre-implement tracking codes for various exceptions generated by themselves, generate corresponding log records at key instruction nodes, and store them in log files for future reference.

[0003] In order to achieve centralized management, in traditional technologies, the platform will also use a service controller with centralized management capabilities, and various online services will upload their generated log files to the service controller, so that the service controller can centralize the log files generated by all online services. In this way, operation and maintenance personnel can analyze the log data from various sources on the service controller to troubleshoot problems. It is not difficult to understand that traditional technology cannot trigger the real-time problem of log upload. Operation and maintenance personnel discover problems based on log files, which lags behind the actual time when the problem occurs.

[0004] In addition to the real-time problem, there is also the problem of data volume. Specifically, the number of online services on each platform is huge, and the log records generated by various online services are naturally massive. A large number of online services frequently submit massive amounts of original log records to the server, which will inevitably cause the service controller to be overwhelmed, and put a test on its operating resources and storage resources. The indirect consequence is that the platform deployment cost remains high.

[0005] Given that traditional technologies have some shortcomings in discovering abnormal events in online services, it is necessary to explore some effective means to capture abnormal events of various online services in a timely and efficient manner in order to quickly discover problems. Summary of the invention

[0006] The purpose of the present application is to solve the above-mentioned problems and to provide a service abnormality alarm method and its corresponding device, equipment, non-volatile readable storage medium, and computer program product.

[0007] According to one aspect of the present application, a service abnormality alarm method is provided, comprising:

[0008] Start the feature detector to monitor the local log records generated by the online service;

[0009] The feature detector intercepts the local log record of the online service and generates a category feature to which the abnormal event represented by the local log record belongs;

[0010] The feature detector periodically sends a reporting event carrying the category feature to the service controller, triggering the service controller to statistically update the total amount of reporting records corresponding to the category feature according to the number of occurrences of the category feature within the corresponding timing period;

[0011] The service controller monitors whether the total amount of reported records of each of the category features exceeds a preset threshold, and generates alarm information corresponding to the category feature when the preset threshold is exceeded.

[0012] According to another aspect of the present application, a service abnormality alarm device is provided, including:

[0013] The service startup module is set to start the feature detector to monitor the local log records generated by the online service;

[0014] A feature generation module, configured to generate a category feature to which the abnormal event represented by the local log record belongs in response to a generation event of a local log record of the online service by the feature detector;

[0015] A reporting processing module, configured to cause the feature detector to send a reporting event carrying the category feature to the service controller, triggering the service controller to statistically update the total amount of reporting records corresponding to the category feature;

[0016] The monitoring and alarm module is configured to monitor, by the service controller, whether the total amount of reported records of each of the category features exceeds a preset threshold, and when the preset threshold is exceeded, generate alarm information corresponding to the category feature.

[0017] According to another aspect of the present application, a service abnormality alarm device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the service abnormality alarm method described in the present application.

[0018] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the service abnormality alarm method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0019] According to another aspect of the present application, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of the method described in any one of the embodiments of the present application are implemented.

[0020] Compared with the prior art, the present application has many technical advantages, including but not limited to: the present application intercepts local log records triggered by abnormal events of online services by setting up feature detectors in online services, extracts category features from local log records, and then periodically obtains the number of occurrences corresponding to each category feature, and submits the number of occurrences associated with the category feature to the service controller for statistics, determines the total number of reported records corresponding to each category feature, and determines whether an alarm is needed based on the total number of reported records. Thus, the division of labor between online services and service controllers is reasonably allocated, and the online service counts the number of occurrences of each category feature in a short period and reports it to the service controller regularly, which greatly reduces the number of requests to be responded to by the service controller. At the same time, it can avoid issuing alarms based on excessively delayed post-event information, and instead use short-period relatively immediate in-event information for alarms. Thus, the processing timeliness and data scale required for abnormal alarms of online services are balanced, so that the operating resources and storage resources required to be consumed by the service controller are lower, making its operating state more robust, and abnormal events are discovered more quickly, with lower deployment and implementation costs, and economies of scale are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of a network architecture of an exemplary deployment environment of the technical solution of the present application;

[0023] Figure 2 A flowchart of an embodiment of the service abnormality alarm method of the present application;

[0024] Figure 3 A schematic diagram of a process of processing local log records by a feature detector in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the flow of interaction between the feature detector and the service controller in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a process for updating the total amount of reported records by the service controller in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a process of distributing feature detectors to a service controller in an embodiment of the present application;

[0028] Figure 7This is a flow chart of the service controller responding to an event viewing instruction in an embodiment of the present application;

[0029] Figure 8 This is a functional block diagram of the service abnormality alarm device of this application;

[0030] Fig. 9 This is a structural diagram of a service abnormality alarm device used in this application. DETAILED DESCRIPTION

[0031] See also Figure 1 The network architecture adopted in the exemplary application scenario of the present application includes a terminal device 80, a front-end server 81 and a service controller 82. The service controller 82 can be used to deploy a service abnormality alarm service and distribute corresponding functional plug-ins such as feature detectors to the online service implemented on the front-end server 81, so as to facilitate data communication with the service abnormality alarm service through the functional plug-ins. When the operation and maintenance user accesses the corresponding page of the service controller 81 from his terminal device 80, he can obtain various information provided by the service abnormality alarm service.

[0032] The service abnormality alarm service of the present application can be implemented by executing the service abnormality alarm method of the present application. Specifically, the service abnormality alarm method of the present application can be implemented as a computer program product, which is installed in a corresponding device such as the service controller. After it is run, the method can be executed by the various functional components of the computer program product to open the service abnormality alarm service.

[0033] See also Figure 2 According to a service abnormality alarm method provided by the present application, in one embodiment thereof, the method comprises the following steps:

[0034] Step S1100: Start a feature detector to monitor local log records generated by online services;

[0035] In one embodiment, through pre-configuration, when various online services in the Internet are started, the feature detectors preset for each online service can be started, and the feature detectors can enter the servo state to receive local log records redirected by the online service. In another embodiment, the feature detectors can be started in advance in the front-end servers deployed by each online service, so that the feature detectors can enter the servo state in advance, which can also achieve the same effect.

[0036] The feature detector can monitor the local log records generated by the online service in a variety of ways. In one embodiment, the local log record output interface of the online service is redirected to the feature detector by modifying the log management module preset by the online service; in another embodiment, the feature extractor is implemented as a hook function (Hook) to hook the code instructions corresponding to the local log records output by the online service. When the code instructions are to be executed, the corresponding business logic of the feature detector is executed; in another embodiment, the feature extractor can also monitor the incremental local log records generated by the corresponding online service in the shared memory or local log file to achieve monitoring. It can be seen that there are many ways to enable the feature detector to obtain the local log records generated by the online service.

[0037] There may be multiple online services, which are deployed in multiple front-end servers respectively, and multiple online services may be deployed in the same front-end server. Each online service may be configured with a dedicated feature detector, or multiple online services may be configured to share a feature detector in the same front-end server.

[0038] The local log record is determined according to the data structure specified by the online service that generates it. For example, in one embodiment, the local log record can organize the corresponding data according to the following data structure:

[0039] {Exception event level; generation time; thread ID; file name; exception instruction line number; message body}

[0040] Different online services usually have different local log record data structures. For feature detectors, they can adapt to the corresponding online services in advance, determine the specific types of local log records that need to be obtained, and perform identification processing.

[0041] Step S1200: intercepting the local log record of the online service by the feature detector, and generating a category feature to which the abnormal event represented by the local log record belongs;

[0042] When the feature detector enters the servo state, it can intercept the local log records generated by the corresponding online service. Each local log record usually indicates that an abnormal event has occurred. Therefore, by parsing the data structure of the local log record, the attribute items and corresponding attribute values ​​therein are obtained. By using the attribute values ​​of some or all of the attribute items, the abnormal event indicated by each local log record can be determined. Based on these attribute values, the corresponding category features of the local log record can be generated to indicate the type of abnormal event.

[0043] When generating the category feature of the abnormal event represented by the local log record based on some attribute values ​​in the local log record, the attribute values ​​corresponding to the attribute items with distinguishing significance for the abnormal event can be selected, and these attribute values ​​are used to form the category data to generate the category feature. The attribute items selected for the category data usually have attribute values ​​that can reflect the uniqueness of the abnormal event represented by the local log record, so that after the corresponding category feature is generated based on the category data, the generated category feature can also obtain the corresponding uniqueness.

[0044] In one embodiment, the categorical data may be the following attribute items: abnormal event level, file name, abnormal instruction line number. It is not difficult to see that in this example, the nature, source and specific location of the abnormal event can be determined through the abnormal event level, file name and abnormal instruction line number, so as to accurately locate the abnormal event and facilitate the rapid investigation of abnormal events. According to this method, it is not difficult to understand that each category feature generated can not only distinguish the nature of the abnormal event, but also distinguish its source and specific location. It does not mean that as long as the nature, source or location of the abnormal event is different, different category features will be generated. Through the same category feature, the nature, source and specific location of the corresponding abnormal event can be determined.

[0045] When generating the corresponding category feature with unique characteristics based on the category data, a hash function can be used to construct the category feature. The hash function can map the category data into a coding string with uniform word length and uniqueness. The coding string can be 128, 256 or 512-bit binary data, and can also be converted into hexadecimal data.

[0046] When a local log record corresponding to an abnormal event arrives for the first time, its category data and its corresponding category characteristics can be determined according to the above process. To facilitate the service controller in subsequent investigation and determination of abnormal events, the category data and category characteristics of the local log record that arrives first can be sent to the service controller as mapping relationship data. The service controller will store this mapping relationship data in its database to facilitate subsequent calls.

[0047] After the service controller has mastered the categorical data and its categorical features corresponding to the abnormal event, when the feature detector subsequently obtains the same categorical data and generates categorical features for it, there is no need to transmit the categorical data to the service controller. Instead, it only needs to upload the mapping relationship data consisting of its categorical features and the number of occurrences obtained by periodic statistics to the service controller.

[0048] Step S1300: the feature detector periodically sends a reporting event carrying the category feature to the service controller, triggering the service controller to update the total amount of reporting records corresponding to the category feature according to the number of occurrences of the category feature in the corresponding timing period;

[0049] In order to avoid frequently sending reports corresponding to each local log record to the service controller and reduce the traffic pressure of the service controller, the feature detector adopts a caching mechanism to open a cache area in the memory of the front-end server where it is located to cache the number of occurrences of each category feature counted within a preset timing period.

[0050] Specifically, after a new category feature appears for the first time, the number of occurrences of local log records belonging to the category feature can be periodically counted, and the total number of occurrences of local log records of the same category feature is accumulated in each timing cycle, thereby obtaining the mapping relationship data between each category feature and its number of occurrences. Each cycle can start accumulating the number of occurrences based on an initial value. When a cycle ends, the number of occurrences of each category feature can be reset to the initial value, such as 0, and then re-counted based on the initial value in the next cycle.

[0051] When a cycle ends, the feature detector can encapsulate the mapping relationship data consisting of each category feature and its corresponding number of occurrences in the same message body, and submit it to the service controller through the corresponding reporting event. In this way, it avoids reporting to the service controller every time a local log record is generated, but periodically reports the periodic number of occurrences corresponding to each category feature to the service controller, which can greatly reduce data interaction with the service controller and reduce the traffic load of the service controller. At the same time, since the category data corresponding to the category feature has been submitted to the service controller in advance, when submitting the number of occurrences corresponding to each category feature to the service controller later, it is no longer necessary to submit the category data of the category feature again. The service controller only needs to query and obtain the corresponding category data based on the category feature provided together with the number of occurrences.

[0052] The service controller is responsible for maintaining an alarm detection table for storing the total amount of reported records of category features of categorical data that have been filed. Specifically, the service controller can store the mapping relationship data between the category feature and its categorical data after receiving it, and then create a data record corresponding to the category feature in the alarm detection table, which contains at least two fields, one is the category field and the other is the event total field. The category field is used to store the category feature, and the event total field is used to store the total amount of reported records of the corresponding category feature. The total amount of reported records can be initialized to a value of 0. In one embodiment, the category feature and its total amount of reported records can be stored in the form of Key-Value based on Redis, that is, the category feature is stored as key domain data, and the total amount of reported records is stored as value domain data. In another embodiment, the data record can also be implemented based on relational data, which can not only contain the category field and the event total field, but also contain a data field for storing the categorical data corresponding to the category feature.

[0053] Each time the service controller receives the category features and their occurrence times periodically submitted by the feature detectors running on each front-end server, it first searches and determines the specific data records in the alarm detection table based on the category features, and then adds the corresponding occurrence times of the category features to the total number of reported records corresponding to the category features in its data records, so that the total number of reported records can be updated, thereby obtaining the latest total number of reported records after the phased update.

[0054] It can be seen that although the total amount of reported records of each category feature is not updated immediately for each local log record, but is updated periodically, the timing period can be set to a reasonable interval, such as 1 minute, 30 seconds, etc., thus achieving a compromise between traffic efficiency and real-time performance. This can avoid the service controller from frequently responding to massive and concurrent local log record-related requests from many online services, greatly reducing the load on the service controller and making the operation of the service controller more robust.

[0055] Step S1400: The service controller monitors whether the total amount of reported records of each category feature exceeds a preset threshold, and generates alarm information corresponding to the category feature when the preset threshold is exceeded.

[0056] In order to identify whether abnormal events corresponding to each category feature need intervention, the service controller can monitor the total amount of reported records of each category feature in the alarm detection table through an independent process or thread, and issue an alarm when an abnormality is monitored.

[0057] Specifically, a threshold corresponding to the triggering alarm is preset, and the alarm detection table is traversed periodically or irregularly through the process or thread. For each category feature in the data record, it is detected whether the corresponding total amount of reported records exceeds the threshold. When it does not exceed the threshold, it indicates that the alarm condition is not met and it is ignored; when it exceeds the threshold, it indicates that the alarm condition is met, so the alarm information corresponding to the category feature can be generated and sent to the preset communication interface.

[0058] In one embodiment, the alarm information may include the category characteristics and the corresponding category data called from the database according to the category characteristics. Of course, it may also include the total number of reported records corresponding to the category characteristics as needed. Since the category data describes the nature, source, location and other information of the abnormal event, the details of the abnormal event can be quickly grasped through the alarm information.

[0059] The communication interface may be an instant messaging interface or an email address, through which the alarm information may be delivered to a corresponding operation and maintenance user, so that the relevant operation and maintenance user may quickly grasp the details of the abnormal event and make a quick disposal.

[0060] In one embodiment, in step S1200, when the feature detector first submits the categorical data and its categorical features corresponding to an abnormal event to the service controller, it can further obtain the operating status information of the corresponding online service, such as its CPU occupancy rate, storage space occupancy rate, etc., and then associate these operating status information with the categorical features and submit them to the service controller for storage in the database for future reference. In step S1400, when it is necessary to encapsulate the alarm information, the operating status information and the categorical data can be encapsulated in the alarm information, so that the operation and maintenance user can not only view the categorical data corresponding to the abnormal event, clarify the nature, source and location of the abnormal event, but also understand the health of the front-end service where the online service is located, so that the information is more comprehensive, and it is convenient to quickly, accurately and efficiently handle abnormal events.

[0061] According to the above embodiments, the present application has many technical advantages, including but not limited to: the present application intercepts local log records triggered by abnormal events of online services by setting a feature detector in the online service, extracts category features from the local log records, and then periodically obtains the number of occurrences corresponding to each category feature, and submits the number of occurrences associated with the category feature to the service controller for statistics, determines the total number of reported records corresponding to each category feature, and determines whether an alarm is needed based on the total number of reported records. Thus, the division of labor between the online service and the service controller is reasonably allocated, and the online service counts the number of occurrences of each category feature in a short period and reports it to the service controller regularly, so that the number of requests to be responded to by the service controller is greatly reduced. At the same time, it can avoid issuing alarms based on excessively delayed post-event information, but use short-term relatively immediate in-event information for alarms. Thus, the processing timeliness and data scale required for abnormal alarms of online services are balanced, so that the operating resources and storage resources required to be consumed by the service controller are lower, making its operating state more robust, and discovering abnormal events more quickly, with lower deployment and implementation costs, and obtaining economies of scale.

[0062] Based on any embodiment of this application, please refer to Figure 3 , the feature detector intercepts the local log record of the online service and generates a category feature to which the abnormal event represented by the local log record belongs, including:

[0063] Step S1210: the feature detector receives the local log record redirected by the online service, parses the data structure of the local log record, extracts attribute values ​​of some attribute items therein to form categorical data;

[0064] In this embodiment, the output object of the local log record of the online service can be configured in advance and redirected to the feature detector corresponding to the online service through configuration, so that the feature detector can receive each local log record output by the online service.

[0065] After the feature detector obtains the local log record, it parses its data structure, and then extracts the attribute values ​​of the corresponding attribute items in the local log record according to the multiple attribute items given in the preset template corresponding to the description of the abnormal event. These attribute values ​​constitute categorical data.

[0066] Step S1220, the feature detector determines whether the attribute value of the predetermined attribute item meets the preset condition, and if not, ignores the local log record; if so, generates the category feature of the abnormal event represented by it according to the category data;

[0067] The local log records output by the online service usually contain different levels of types. Therefore, the attribute value of the corresponding level is usually recorded in an attribute item of each local log record that is specifically used to indicate the level of abnormal events. For example, the attribute value can be FATAL, ERROR, WARN, INFO, DEBUG, TRACE, and ALL, indicating the levels of fatal, error, warning, information, debugging, tracing, and general abnormal events, respectively. However, only some types of local log records may need targeted alerts, such as the three types of FATAL, ERROR, and WARN. In view of this situation, the feature detector pre-sets these attribute values ​​as predetermined attribute values, and then, for each local log record, determines whether the attribute value of the corresponding attribute item is one of the predetermined attribute values. If not, the local log record can be ignored and is not used as a basis for counting the number of occurrences of the corresponding category feature. If not, the local log record can be used as the basis for counting the number of occurrences of its corresponding category feature. According to the relevant process of this application, the category data is extracted from the local log record to generate the corresponding category feature, and then the number of occurrences of the category feature is statistically updated.

[0068] Through this step, local log records can be screened to avoid generating and reporting category features for each local log record. Instead, key type information can be focused on, thereby improving the processing efficiency of the feature detector and making the statistical information more accurate.

[0069] Step S1230: The feature detector stores the local log record in a log file of the online service.

[0070] Considering that the online service needs to maintain its locally stored log files, and the feature detector intercepts the local log records of the online service through redirection, after completing the screening and subsequent processing of the local log records, the intercepted local log records can be stored back in the log file of the online service to avoid damaging the log file of the online service.

[0071] According to the above embodiments, the feature detector obtains the local log records of the online service based on redirection, processes them and then transmits them back for storage in the log file of the online service. This will not damage the original business logic of the online service, and the online service does not need to implement the business logic of interacting with the service controller itself. It only needs to load the standardized feature detector to filter the local log files, thereby further realizing the reporting of abnormal events of specific levels. This is very efficient and convenient, and is conducive to standardized implementation.

[0072] Based on any embodiment of this application, please refer to Figure 4, after generating the category features of the abnormal events represented by the category data, it includes:

[0073] Step S1221, the feature detector determines whether the categorical data appears for the first time, and if it appears for the first time, starts a periodic timer to count the number of occurrences of the categorical data within a preset period, and submits an initialization instruction to the service controller, wherein the initialization instruction carries the categorical data and its categorical feature;

[0074] After extracting the categorical data of a local log record each time and generating the corresponding categorical features based on the categorical data, first determine whether the categorical data appears for the first time. This can be achieved in a variety of ways. In one embodiment, the feature detector can set a mapping table in the local cache to store the mapping relationship data between the categorical data and its categorical features. When the mapping table has such mapping relationship data, it can be confirmed that the corresponding categorical data does not appear for the first time. When the mapping table does not have such mapping relationship data, it can be confirmed that the categorical data appears for the first time. In another embodiment, when the service controller stores the mapping relationship data between the categorical features and their categorical data, a query request can be submitted to the service controller, and the service controller queries whether it has stored the corresponding categorical data and the mapping relationship data between the categorical features and returns the result. When the service controller does not have the corresponding mapping relationship data, the corresponding categorical data can be regarded as appearing for the first time.

[0075] For the first appearance of categorical data, the feature detector starts the corresponding cycle timer for it and calculates the corresponding timing cycle according to the corresponding duration of the predetermined cycle, such as 1 minute or 30 seconds. When a timing cycle ends, the cycle timer can be reset to continue timing the next cycle.

[0076] Within a timing cycle corresponding to the periodic timer, the feature detector will calculate the number of occurrences of the corresponding local log records for the same category feature. That is, after the corresponding category data appears for the first time, the feature detector will count the number of occurrences of the category feature generated according to the category data within the corresponding timing cycle. In fact, it is to count the periodic total number of local log records corresponding to the same category feature.

[0077] After confirming that the categorical data appears for the first time, the feature detector also sends an initialization instruction to the service controller. The initialization instruction carries the categorical data that appears for the first time and the category features generated therefrom, so that the two constitute mapping relationship data.

[0078] It is not difficult to understand that if the category data of the local log record obtained each time does not appear for the first time, there is no need to submit the initialization instruction, but only to accumulate and increase the number of occurrences of the local log record.

[0079] In one embodiment, the initialization instruction may carry the running status information of the online service, so that the service controller can associate the category data of the storage category feature with the running status information for query.

[0080] Step S1222: The service controller creates a data record with the category feature as an index in the database in response to the initialization instruction, associates the category feature with its category data, and initializes the total amount of reported records corresponding to the category feature to an initial value;

[0081] After receiving the initialization instruction, the service controller responds, parses the mapping relationship data, and then creates a data record in the database for correspondingly storing the mapping relationship data, so as to store the category data and its category characteristics in the database, thereby realizing the filing of the category data corresponding to the category characteristics, which can be queried and called later.

[0082] The action of storing the categorical data in the database means that the service controller receives a new category feature and needs to start counting the total number of reported records corresponding to the category feature. Therefore, correspondingly, the category feature is also associated in the database, and its corresponding total number of reported records is stored, and the total number of reported records is initialized to an initial value, such as 0. In the embodiments disclosed in the foregoing text of this application, the mapping relationship between the category feature and its total number of reported records can be stored in the form of a key-value pair.

[0083] Step S1223, in response to the timing arrival event of the periodic timer, the number of occurrences obtained within the timing period of the periodic timer and the category feature form mapping relationship data, generate the reporting event, and include the mapping relationship data in the reporting event.

[0084] When the periodic timer reaches the timing, a timing arrival event will be triggered. In response to the timing arrival event, the mapping relationship data between the category features and their occurrence times obtained by statistics in the corresponding timing period can be encapsulated and encapsulated into a corresponding message body. Then, a reporting event is constructed corresponding to the message body, and the reporting event is sent to the service controller. The message body is submitted to the service controller through the reporting event, so that the service controller obtains each category feature and its occurrence times in the corresponding period, which is used to update the total number of reporting records corresponding to each category feature.

[0085] According to the above embodiments, it can be known that the feature detector only needs to submit the category data to the service controller when the category data of each category feature appears for the first time. At other times, it only needs to periodically report the number of occurrences corresponding to the category features so that the service controller can quickly count the total number of reporting records corresponding to each category feature. On the online service side, there is no need to transmit data to the service controller frequently and in large quantities, which occupies less bandwidth. Since the service controller obtains less data and lower frequency from each online service side, its traffic occupancy is also very low. The service controller runs more efficiently and robustly without affecting the discovery of abnormal events. It takes into account both the timely reporting of abnormal events and the balance of traffic load. Moreover, since the category data corresponding to each category feature is also filed with the service controller, it does not affect the subsequent efficient investigation of abnormal events of massive online services.

[0086] Based on any embodiment of this application, please refer to Figure 5 The feature detector periodically sends a reporting event carrying the category feature to the service controller, triggering the service controller to statistically update the total amount of reporting records corresponding to the category feature according to the number of occurrences of the category feature within the corresponding timing period, including:

[0087] Step S1310: The service controller responds to the reported event by parsing the mapping relationship data carried therein to obtain the corresponding category features and their corresponding occurrence times within a single timing cycle;

[0088] When the service control receives a reporting event submitted by a feature detector corresponding to any online service, it responds to the reporting event and parses the mapping relationship data carried therein. As mentioned above, the mapping relationship data includes each category feature and its corresponding number of occurrences within a period, thereby obtaining each category feature and its corresponding number of occurrences within a single timing period.

[0089] Step S1320: The service controller searches the database for data records corresponding to the category feature, adds the number of occurrences to the total number of reported records in the data record, and updates the total number of reported records.

[0090] The service controller stores the category features and their corresponding total number of reported records in a database. Therefore, after obtaining the mapping relationship data of new category features and their occurrence times, the corresponding data records are queried and determined from the database based on the category features. Then, based on the total number of reported records in the data records of the category features, the corresponding number of occurrences of the category features are accumulated to complete the update of the total number of reported records of the category features.

[0091] According to the above embodiments, the service controller can quickly find the corresponding data records according to the category characteristics, and timely accumulate the corresponding number of occurrences of the category characteristics to the total number of reporting records corresponding to the category characteristics, so as to achieve fast and lightweight updates. When faced with a large number of online services concurrently submitting a large number of reporting events, it can reflect efficiency advantages.

[0092] Based on any embodiment of this application, please refer to Figure 6 , before starting the feature detector to monitor local log records generated by online services, including:

[0093] Step S2100: The service controller responds to the deployment completion notification message corresponding to the online service and sends the feature detector and its identity token to the online service;

[0094] Various online services can be implemented through different development frameworks. Therefore, the feature detectors corresponding to different development frameworks can be standardized and then distributed to each online service by the service controller.

[0095] When the online service is deployed on the corresponding front-end server, a distribution request can be submitted to the service controller, and the distribution request includes the corresponding development framework identification information. After receiving the distribution request, the service controller searches for the resource file of the corresponding feature detector based on the development framework identification information, and at the same time, generates an identity token with a unique identity identifier, and sends the resource file and its corresponding identity token to the corresponding online service.

[0096] Step S2200: The online service configures the feature detector as a synchronization plug-in that is started in response to the online service, and configures the identity token as necessary information required for the feature detector to communicate data with the service controller.

[0097] When the online service receives the resource file and identity token corresponding to the feature detector, the identity token is cached and the resource text is configured as a synchronization plug-in that is started in response to the online service startup, thereby completing the configuration.

[0098] After configuration, each time the online service is started, the pre-configured resource file will be run and the pre-configured feature detector will be invoked accordingly. During data communication with the service controller, the feature detector will carry the identity token in its request or instruction each time. The service controller will verify the legitimacy of the communication of the feature detector with the identity token each time, and will only accept legal requests or instructions and refuse to respond to illegal requests or instructions, thereby protecting the service controller from illegal attacks and ensuring network security.

[0099] According to the above embodiments, feature detectors can be implemented in a standardized manner and can be centrally distributed by service controllers, which is conducive to large-scale platforms to centrally configure massive online services in distributed architectures, making the installation and updating of feature detectors more efficient, and can also ensure network communication security through identity tokens, making the operation of service controllers more robust.

[0100] Based on any embodiment of this application, please refer to Figure 7 The service controller monitors whether the total amount of reported records of each of the category features exceeds a preset threshold value. When the preset threshold value is exceeded, after generating the alarm information corresponding to the category feature, it includes:

[0101] Step S3100: The service controller responds to the event viewing instruction to obtain the characteristics of each category and its category data and the total amount of reported records in the database to form corresponding data samples;

[0102] The operation and maintenance user can view the overall situation of abnormal events of the massive online services of the entire large-scale Internet platform by accessing the corresponding page provided by the service controller. Specifically, through the page, the operation and maintenance user can trigger an event viewing instruction to be sent to the service controller. After the service controller receives the event viewing instruction, it responds to the instruction and can call the specific data in the database, which may specifically include the mapping relationship data between the category features in the alarm detection table and the total amount of their reported records, and may also include the mapping relationship data between the category features and their corresponding categorical data, that is, the mapping relationship data between each category feature and its total amount of reported records, and the categorical data, and construct each category feature and its corresponding total amount of reported records into a corresponding data sample. It is not difficult to understand that by including categorical data in the construction of the data sample, the semantics of the data sample can be enriched, and it can ensure that subsequent clustering obtains more accurate clustering results.

[0103] Step S3200: The service controller clusters the data samples to obtain multiple clusters, and determines a cumulative value of the total number of reported records of the data samples included in each cluster as the total number of cluster events;

[0104] Considering that some features of different categories may belong to different online services, but may have similarities due to the file names, and the total amount of reported records may also show certain similarities, clustering algorithms can be applied to cluster these data samples in order to comprehensively examine the overall classification characteristics of massive abnormal events from the perspective of the entire platform.

[0105] In one embodiment, in order to facilitate subsequent clustering processing, each item of information in each data sample can be encoded accordingly to obtain a corresponding encoding vector, and then the deep semantic information of the encoding vector of each data sample can be extracted through features that have been pre-trained to convergence to obtain its corresponding semantic vector, and all data samples can be clustered based on the semantic vectors of each data sample.

[0106] When performing clustering, any mature clustering algorithm can be used for implementation, including but not limited to any of the following: K-Means clustering algorithm, mean shift clustering algorithm, density-based clustering algorithm (DBSCAN), maximum expectation clustering algorithm based on Gaussian mixture model, agglomerative hierarchical clustering algorithm, graph group detection clustering algorithm, etc.

[0107] The K-Means clustering algorithm needs to select some clusters first, and randomly initialize the core sample points of each cluster as the center point. The sample point can be a word pair and its similarity. The center point is the position with the same length as the vector of each sample point. Then, calculate the distance from each sample point to the center point, and the sample point is divided into the cluster to which the center point is closest. Then, recalculate and determine the center point of each cluster. Repeat the above steps until the center point of each cluster does not change more than the preset range after each iteration. You can also randomly initialize the center point multiple times, and then select the one with the best running result.

[0108] The mean shift clustering algorithm is an algorithm based on sliding windows, which uses sliding windows to find dense areas of sample points. This is a centroid-based algorithm that locates the center point of each cluster by updating the candidate points of the center point to the mean of the points in the sliding window. Then, similar windows of these candidate windows are removed to finally form a set of center points and corresponding clusters.

[0109] Density-based clustering algorithm (DBSCAN), similar to the mean shift clustering algorithm, is also a density-based clustering algorithm. It first determines an adjacent range and a preset number threshold, starting from an arbitrary sample point that has not been visited, taking this sample point as the center point, and detecting whether the number of sample points contained in the adjacent range is greater than or equal to the preset number threshold. If it is greater than or equal to the preset number threshold, the sample point is marked as the center point, otherwise it is marked as a noise node. Repeat the above steps. If a noise node exists in the adjacent range of a certain center point, the noise node is marked as an edge node, otherwise it is still a noise node. Repeat the above process until all sample points have been visited.

[0110] The maximum expectation (EM) clustering algorithm based on the Gaussian mixture model (GMM) first selects the number of clusters (similar to K-Means) and randomly initializes the Gaussian distribution parameters (mean and variance) of each cluster. You can also observe the data first to give a relatively accurate mean and variance. Then, given the Gaussian distribution of each cluster, calculate the probability that each sample point belongs to each cluster. The closer a sample point is to the center of the Gaussian distribution, the more likely it is to belong to the cluster. Then, based on these probabilities, the Gaussian distribution parameters are calculated to maximize the probability of the sample point. These new parameters can be calculated using the weighted probabilities of the sample points, and the weight is the probability that the sample point belongs to the cluster. Then, repeat the first two steps until the changes in the iteration do not exceed the preset range, and the clustering is completed.

[0111] Agglomerative hierarchical clustering algorithms are divided into two categories: top-down and bottom-up. Hierarchical Agglomerative Clustering (HAC) is a bottom-up clustering algorithm. HAC first considers each sample point as a single cluster, and then calculates the distance between all clusters to merge clusters until all clusters are aggregated into one cluster.

[0112] The first step of the Graph Community Detection clustering algorithm is to initially assign each sample point regarded as a vertex to its own community, and then calculate the modularity M of the entire network. In the second step, if two communities are merged together, the algorithm calculates the resulting modularity change ΔM, which is essentially to take the community pair with the largest increase in ΔM and then merge them. In the third step, the new modularity M is calculated for this cluster and recorded. Then, the first and second steps are repeated, each time the community pairs are merged, so that the maximum gain of ΔM is finally obtained, and the new clustering pattern and its corresponding modularity score M are recorded.

[0113] According to the above introduction to each optional clustering algorithm, it can be known that by using any feasible clustering algorithm, clustering each word pair based on the similarity of each word pair, one or more similar clusters can be determined, and these similar clusters are composed of multiple data samples. Since each data sample establishes a de facto edge connection relationship based on similarity, each similar cluster obtained by clustering is generally composed of data samples with the same semantics.

[0114] Step S3300: sort each cluster according to the total number of cluster events, generate a mapping relationship list of clusters and their total number of cluster events, and respond to the event viewing instruction.

[0115] Each cluster has its corresponding total number of cluster events. The larger the total number of cluster events, the more significant the corresponding problem is. Therefore, according to the principle of urgency, the clusters are sorted according to the total number of cluster events, and a mapping relationship list can be obtained. In the mapping relationship list, the mapping relationship data between the cluster identifier and the total number of cluster events are stored, and then the mapping relationship list is pushed to the operation and maintenance user to complete the response to the event viewing instruction. The operation and maintenance user can then conduct an overall problem investigation after the browser parses and displays the mapping relationship list. Furthermore, when the operation and maintenance user clicks on the identifier of any cluster, the details of each data sample in the cluster can be further pushed to the user, displaying the category characteristics of each data sample and its corresponding category data as well as the corresponding online service operation status information, etc., so that the operation and maintenance user can analyze the problems corresponding to the abnormal events in the entire Internet platform from shallow to deep and deal with them.

[0116] According to the above embodiments, by utilizing the category features of the present application and its category data and the number of reported records to provide rich initial semantics, with the help of clustering algorithms, macro-clustering of abnormal events reported by massive online services can be achieved. By clustering across online services, the associations between abnormal events of different sources, locations, and properties can be mined, so that operation and maintenance users can quickly obtain an overall overview of abnormal events on the entire Internet platform, improve operation and maintenance efficiency, and facilitate rapid troubleshooting of abnormalities.

[0117] See also Figure 8 According to one aspect of the present application, a service abnormality alarm device is provided, comprising a service startup module 1100, a feature generation module 1200, a reporting processing module 1300, and a monitoring alarm module 1400, wherein: the service startup module 1100 is configured to start a feature detector for monitoring local log records generated by online services; the feature generation module 1200 is configured to generate a category feature to which the abnormal event represented by the local log record belongs in response to the generation event of the local log record of the online service by the feature detector; the reporting processing module 1300 is configured to send a reporting event carrying the category feature to a service controller by the feature detector, triggering the service controller to statistically update the total amount of reporting records corresponding to the category feature; the monitoring alarm module 1400 is configured to monitor by the service controller whether the total amount of reporting records of each category feature exceeds a preset threshold, and when the preset threshold is exceeded, generate alarm information corresponding to the category feature.

[0118] Based on any embodiment of the present application, the feature generation module 1200 includes: a record parsing unit, configured for the feature detector to receive the local log record redirected and output by the online service, parse the data structure of the local log record, extract attribute values ​​of some attribute items therein to form categorical data; a record screening unit, configured for the feature detector to determine whether the attribute value of a predetermined attribute item meets a preset condition, if not, ignore the local log record; if so, generate a category feature of the abnormal event represented by it according to the categorical data; a local storage unit, configured for the feature detector to store the local log record in the log file of the online service.

[0119] Based on any embodiment of the present application, the feature generation module 1200 also includes: an initial preparation unit, configured for the feature detector to determine whether the categorical data appears for the first time. If it appears for the first time, start a periodic timer to count the number of occurrences of the categorical data within a preset period, and submit an initialization instruction to the service controller, carrying the categorical data and its categorical feature in the initialization instruction; an initial processing unit, configured for the service controller to respond to the initialization instruction, create a data record indexed by the categorical feature in the database, associate the categorical feature with its categorical data, and initialize the total number of reported records corresponding to the categorical feature to an initial value; a periodic accumulation unit, configured to respond to a timing arrival event of the periodic timer, form a mapping relationship data between the number of occurrences obtained within the timing period of the periodic timer and the categorical feature, generate the reporting event, and include the mapping relationship data in the reporting event.

[0120] Based on any embodiment of the present application, the reporting processing module 1300 includes: a reporting parsing unit, configured for the service controller to respond to the reporting event, parse the mapping relationship data carried therein, and obtain the corresponding category characteristics and their corresponding number of occurrences within a single timing cycle; a total updating unit, configured for the service controller to search the database for the data record corresponding to the category characteristics, add the number of occurrences to the total number of reporting records in the data record, and update the total number of reporting records.

[0121] Based on any embodiment of the present application, the initialization instruction also carries the running status information of the online service, and the categorical data includes attribute values ​​for describing the nature, source and / or location of the abnormal event.

[0122] Based on any embodiment of the present application, the service abnormality alarm device of the present application further includes: a distribution processing module, configured so that the service controller responds to the deployment completion notification message corresponding to the online service, and sends the feature detector and its identity token to the online service; a distribution configuration module, configured so that the online service configures the feature detector as a synchronization plug-in started in response to the online service, and configures the identity token as the necessary information required for the feature detector to communicate data with the service controller.

[0123] On the basis of any embodiment of the present application, the service abnormality alarm device of the present application further includes: a viewing response module, which is configured for the service controller to respond to an event viewing instruction, obtain the characteristics of each category in the database and its category data and the total amount of reported records, and constitute corresponding data samples; a clustering processing module, which is configured for the service controller to cluster the data samples, obtain multiple clusters, and determine the cumulative value of the total amount of reported records of the data samples contained in each cluster as the total amount of cluster events; an information output module, which is configured to sort each cluster according to the total amount of cluster events, generate a mapping relationship list of clusters and their total amount of cluster events, and respond to the event viewing instruction.

[0124] Another embodiment of the present application also provides a service abnormality alarm device. Fig. 9 As shown, a schematic diagram of the internal structure of a service abnormality alarm device. The service abnormality alarm device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable non-volatile readable storage medium of the service abnormality alarm device stores an operating system, a database, and a computer-readable instruction. The database may store an information sequence. When the computer-readable instruction is executed by the processor, the processor may implement a service abnormality alarm method.

[0125] The processor of the service abnormality alarm device is used to provide computing and control capabilities to support the operation of the entire service abnormality alarm device. The memory of the service abnormality alarm device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the service abnormality alarm method of the present application. The network interface of the service abnormality alarm device is used to connect and communicate with the terminal.

[0126] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the service abnormality alarm device to which the present application scheme is applied. The specific service abnormality alarm device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0127] In this embodiment, the processor is used to execute Figure 8 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the service abnormality alarm device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0128] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service abnormality alarm method of any embodiment of the present application.

[0129] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0131] To sum up, the present application can balance the processing time and data scale required for online service abnormal alarms, reduce the operating resources and storage resources required for the service controller, make its operating state more robust, discover abnormal events more quickly, reduce deployment costs, and achieve economies of scale.

Claims

1. A service abnormality alarm method, characterized in that: include: Start the feature detector preset for the online service to monitor the local log records generated by the online service; The feature detector intercepts the local log record of the online service and generates the category feature to which the abnormal event represented by the local log record belongs, including: the feature detector determines whether the category data extracted from the local log record corresponding to the category feature appears for the first time; if it appears for the first time, starts a periodic timer to count the number of occurrences of the category data within a preset period, and submits an initialization instruction to the service controller, in which the category data and its category feature are carried; the service controller responds to the initialization instruction by creating a data record indexed by the category feature in a database, so that the category feature is associated with its category data, and initializes the total number of reported records corresponding to the category feature to an initial value; responds to the timing arrival event of the periodic timer, forms mapping relationship data between the number of occurrences obtained within the timing period of the periodic timer and the category feature, generates a reporting event, and includes the mapping relationship data in the reporting event; The feature detector periodically sends a reporting event carrying the category feature to the service controller, triggering the service controller to statistically update the total amount of reporting records corresponding to the category feature according to the number of occurrences of the category feature in the corresponding timing cycle, including: the service controller responds to the reporting event, parses the mapping relationship data carried therein, obtains the corresponding category feature and its corresponding number of occurrences in a single timing cycle; the service controller searches for the data record corresponding to the category feature from the database, accumulates the number of occurrences to the total amount of reporting records in the data record, and updates the total amount of reporting records; The service controller monitors whether the total amount of reported records of each of the category features exceeds a preset threshold, and generates alarm information corresponding to the category feature when the preset threshold is exceeded.

2. The service abnormality alarm method according to claim 1, characterized in that: The feature detector intercepts the local log record of the online service and generates a category feature to which the abnormal event represented by the local log record belongs, including: The feature detector receives the local log record output by the online service redirection, parses the data structure of the local log record, extracts the attribute values ​​of some attribute items therein to form categorical data; The feature detector determines whether the attribute value of the predetermined attribute item meets the preset condition. If not, the local log record is ignored; if satisfied, the category feature of the abnormal event represented by it is generated according to the category data; The feature detector stores the local log record in a log file of the online service.

3. The service abnormality alarm method according to claim 1, characterized in that: The initialization instruction also carries the running status information of the online service, and the categorical data includes attribute values ​​for describing the nature, source and / or location of the abnormal event.

4. The service abnormality alarm method according to any one of claims 1 to 3, characterized in that: Before starting the feature detector to monitor local log records generated by online services, include: The service controller sends the feature detector and its identity token to the online service in response to a deployment completion notification message corresponding to the online service; The online service configures the feature detector as a synchronization plug-in started in response to the online service, and configures the identity token as necessary information required for the feature detector to perform data communication with the service controller.

5. The service abnormality alarm method according to any one of claims 1 to 3, characterized in that: The service controller monitors whether the total amount of reported records of each of the category features exceeds a preset threshold value. When the preset threshold value is exceeded, after generating the alarm information corresponding to the category feature, the following steps are included: The service controller responds to the event viewing instruction to obtain the characteristics of each category and its category data and the total amount of reported records in the database to form corresponding data samples; The service controller clusters the data samples to obtain a plurality of clusters, and determines a cumulative value of the total number of reported records of the data samples included in each cluster as the total number of cluster events; The clusters are sorted according to the total amount of cluster events, a mapping relationship list of clusters and their total amount of cluster events is generated, and the event viewing instruction is responded to.

6. A service abnormality alarm device, characterized in that: include: A service startup module, configured to start a feature detector preset for the online service, for monitoring local log records generated by the online service; The feature generation module is configured to generate a category feature to which the abnormal event represented by the local log record belongs in response to the generation event of the local log record of the online service by the feature detector, including: the feature detector determines whether the category data extracted from the local log record corresponding to the category feature appears for the first time, and if it appears for the first time, starts a periodic timer to count the number of occurrences of the category data within a preset period, and submits an initialization instruction to the service controller, in which the category data and its category feature are carried in the initialization instruction; the service controller creates a data record indexed by the category feature in a database in response to the initialization instruction, associates the category feature with its category data, and initializes the total number of reported records corresponding to the category feature to an initial value; in response to the timing arrival event of the periodic timer, forms mapping relationship data between the number of occurrences obtained within the timing period of the periodic timer and the category feature, generates a reporting event, and includes the mapping relationship data in the reporting event; The reporting processing module is configured to send a reporting event carrying the category feature to the service controller by the feature detector, triggering the service controller to statistically update the total amount of reporting records corresponding to the category feature, including: the service controller responds to the reporting event, parses the mapping relationship data carried therein, obtains the corresponding category feature and its corresponding number of occurrences within a single timing cycle; the service controller searches the database for the data record corresponding to the category feature, accumulates the number of occurrences to the total amount of reporting records in the data record, and updates the total amount of reporting records; The monitoring and alarm module is configured to monitor, by the service controller, whether the total amount of reported records of each of the category features exceeds a preset threshold, and when the preset threshold is exceeded, generate alarm information corresponding to the category feature.

7. A service abnormality alarm device, comprising a central processing unit and a memory, characterized in that: The central processor is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A non-volatile readable storage medium, characterized in that: It stores a computer program in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method according to any one of claims 1 to 5 are executed.

9. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, performs the steps of the method according to any one of claims 1 to 5.

Images