Log pushing method and device

Abstract

The invention discloses a log pushing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: acquiring at least one piece of configuration information corresponding to scene information; wherein the configuration information comprises a plurality of data fields, a verification rule, a sampling proportion and a pushing condition; obtaining a plurality of log records in the statistical time period; for each piece of configuration information, generating a screening log record corresponding to each log record according to a plurality of data fields corresponding to the configuration information; according to a verification rule corresponding to the configuration information, filtering the plurality of screened log records to obtain a plurality of filtered log records; determining a plurality of to-be-pushed log records from the plurality of filtered log records according to a sampling proportion corresponding to the configuration information; and pushing the plurality of to-be-pushed log records according to a pushing condition corresponding to the configuration information. According to the embodiment, the working pressure of the log analysis server can be relieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log pushing method and device. Background Art

[0002] Logs are used to record the hardware, software, and system operation status in the system. Users can use logs to check the causes of system operation failures or find traces left by attackers when attacked. Logs can exist on security acceleration devices, application servers, and database servers. Usually, all logs on each device and server are pushed to the log analysis server for more detailed log analysis and attack prediction. However, there are too many logs to be collected, which puts a lot of pressure on the log analysis server. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a log pushing method and device, which can reduce the working pressure of a log analysis server.

[0004] In a first aspect, an embodiment of the present invention provides a log push method, comprising:

[0005] Obtain at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios, and push conditions;

[0006] Get multiple log records within the statistical period;

[0007] For each piece of configuration information, according to the multiple data fields corresponding to the configuration information, a filtering log record corresponding to each log record is generated; according to the verification rules corresponding to the configuration information, multiple filtering log records are filtered to obtain multiple filtering log records; according to the sampling ratio corresponding to the configuration information, multiple log records to be pushed are determined from the multiple filtering log records; according to the push conditions corresponding to the configuration information, the multiple log records to be pushed are pushed.

[0008] Optionally, obtaining at least one piece of configuration information corresponding to the scene information includes:

[0009] In response to the scene information indicating that an abnormal event occurs in the current scene, determining at least one event type corresponding to the abnormal event;

[0010] Configuration information corresponding to the scene information is determined according to configuration information corresponding to at least one event type.

[0011] Optionally, determining configuration information corresponding to the scene information according to configuration information corresponding to at least one event type includes:

[0012] Determine the configuration information corresponding to each event type;

[0013] Determine the configuration information corresponding to the normal event type;

[0014] The configuration information corresponding to each event type and the configuration information corresponding to the normal event type are determined as the configuration information corresponding to the scene information.

[0015] Optionally, obtaining at least one piece of configuration information corresponding to the scene information includes:

[0016] In response to the scene information indicating that no abnormal event occurs in the current scene, the configuration information corresponding to the normal event type is determined as the configuration information corresponding to the scene information.

[0017] Optionally, the configuration information also includes: output format and storage server;

[0018] According to the push conditions corresponding to the configuration information, multiple log records to be pushed are pushed, including:

[0019] According to the output format, multiple log records to be pushed are converted into different formats;

[0020] According to the push conditions corresponding to the configuration information, push processing is performed on the converted multiple log records to be pushed, so as to push the converted multiple log records to be pushed to the storage server.

[0021] Optionally, after the multiple log records to be pushed are pushed according to the push conditions corresponding to the configuration information, the method further includes:

[0022] In response to failure in pushing the plurality of log records to be pushed, determining the number of failures corresponding to the plurality of log records to be pushed;

[0023] In response to the number of failures being less than the number threshold, re-pushing the plurality of log records to be pushed;

[0024] In response to the number of failures being not less than a number threshold, generating alarm information for the plurality of log records to be pushed.

[0025] In a second aspect, an embodiment of the present invention provides a log pushing device, including:

[0026] An information acquisition module, used to acquire at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions;

[0027] The record acquisition module is used to obtain multiple log records within the statistical period;

[0028] The log push module is used to generate a filtering log record corresponding to each log record according to multiple data fields corresponding to the configuration information for each configuration information; filter multiple filtering log records according to the verification rules corresponding to the configuration information to obtain multiple filtering log records; determine multiple log records to be pushed from the multiple filtering log records according to the sampling ratio corresponding to the configuration information; and push the multiple log records to be pushed according to the push conditions corresponding to the configuration information.

[0029] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0030] one or more processors;

[0031] a storage device for storing one or more programs,

[0032] When one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method of any of the above embodiments is implemented.

[0034] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor.

[0035] An embodiment of the above invention has the following advantages or beneficial effects: selecting at least one configuration information corresponding to the scene information, and processing and pushing the log record according to the configuration information. The configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions. During the log record processing, it is only necessary to retain the values ​​of each data field corresponding to the configuration information, and select the log record according to the verification rules and the adoption ratio. Compared with the method of pushing the full amount of logs, it can reduce the pressure on the log analysis server.

[0036] In addition, different configuration information is set according to different scenarios of the system. The pushed log records are more in line with business needs, and the information in the log records is more targeted, which is convenient for subsequent log analysis and attack prediction.

[0037] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0039] Figure 1It is a schematic diagram of a process of a log pushing method provided by an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a process of a log pushing method provided by another embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of a process of a log pushing method provided by another embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of the structure of a log push system provided by an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of a process of a log pushing method provided by another embodiment of the present invention;

[0044] Figure 6 It is a structural schematic diagram of a log pushing device provided by an embodiment of the present invention;

[0045] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0047] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of the embodiments of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0048] Figure 1 FIG. 1 is a schematic diagram of a log push method process provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0049] Step 101: Obtain at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions.

[0050] Scenario information is used to characterize the scenario characteristics of the target system. Scenario information can characterize whether an abnormal event occurs or not in the current scenario. An abnormal event corresponds to at least one event type. Event types can be: BOT (robot) attack, DDoS (Distributed Denial of Service) attack, overseas attack, page abnormality, etc.

[0051] The scenario information can be determined in a variety of ways. For example, if a warning message about a page anomaly is received for the target system, the scenario information is determined to indicate that an abnormal event has occurred in the current scenario, and the event type corresponding to the abnormal event is page anomaly. For another example, if the user has activated BOT attack monitoring and DDoS attack monitoring for the target system, the scenario information is determined to indicate that an abnormal event has occurred in the current scenario, and the event type corresponding to the abnormal event is BOT attack and DDoS attack.

[0052] Configuration information can be flexibly set according to needs. Each configuration information includes: multiple data fields, verification rules, sampling ratios, and push conditions. The original log record may correspond to multiple fields. Some fields are meaningless for log analysis and do not need to be pushed to the log analysis server. The data field is the field in the log record that needs to collect the corresponding value.

[0053] Verification rules are used to filter log records to collect filtered log records that meet the verification rules. Sampling ratio is used to filter a certain proportion of log records to be pushed from multiple filtered log records.

[0054] Step 102: Obtain multiple log records within the statistical period.

[0055] The statistical period can be flexibly set according to needs. The statistical period can correspond to the statistical year, month, week, day, time period, etc.

[0056] Step 103: for each piece of configuration information, generate a screening log record corresponding to each log record according to multiple data fields corresponding to the configuration information; and filter the multiple screening log records according to the verification rule corresponding to the configuration information to obtain multiple filtering log records.

[0057] The fields corresponding to the filtered log records are the same as the data fields corresponding to the configuration information. For example, the fields corresponding to the log records are: request ID, source server IP address, client IP address, country to which the client IP address belongs, number of bytes requested by the client, URI requested by the client, Http protocol requested by the client, size of HTTP response Body, HTTP status code, etc. The data fields corresponding to the configuration information are: request ID, source server IP address, client IP address, number of bytes requested by the client, HTTP status code, then the fields corresponding to the filtered log records are: request ID, source server IP address, client IP address, number of bytes requested by the client, HTTP status code. Other fields other than the data fields corresponding to the configuration information do not need to be collected.

[0058] According to the verification rules corresponding to the configuration information, multiple filtering log records are filtered to obtain multiple filtering log records. Each filtering log record complies with the verification rules. The verification rules can be described by Boolean expressions or expressions that other servers can parse. For example, the verification rule can be {"key":"zoneName","operator":"eq","value":"example.com"}, which represents the push of log data with a domain name equal to example.com. Among them, "key":"zoneName" represents the field name as "zoneName". "operator":"eq" represents the filtering condition as equal to. The filtering conditions can also be: "lt", "gt", "contains", etc. Among them, "lt" represents less than, "gt" represents greater than, and "contains" represents inclusion.

[0059] Step 104: according to the sampling ratio corresponding to the configuration information, a plurality of log records to be pushed are determined from the plurality of filtered log records; according to the push conditions corresponding to the configuration information, the plurality of log records to be pushed are pushed.

[0060] The sampling ratio may be in the form of a percentage. For example, if the sampling ratio is 10% and the number of filtered log records is 100, 10 log records to be pushed are selected from the multiple filtered log records.

[0061] The push conditions can be set according to specific needs. The push conditions can be set to preset time points, preset numbers, etc. When the preset time point is reached, multiple log records to be pushed are pushed at a scheduled time. Or when the number of log records to be pushed exceeds the preset number, multiple log records to be pushed are pushed.

[0062] In the scheme of the embodiment of the present invention, at least one configuration information corresponding to the scene information is selected, and the log record is processed and pushed according to the configuration information. The configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions. During the log record processing, it is only necessary to retain the values ​​of each data field corresponding to the configuration information, and select the log record according to the verification rules and the adoption ratio. Compared with the method of pushing the log in full, it can reduce the pressure on the log analysis server.

[0063] In addition, different configuration information is set according to different scenarios of the system. The pushed log records are more in line with business needs, and the information in the log records is more targeted, which is convenient for subsequent log analysis and attack prediction.

[0064] In one embodiment of the present invention, after the push processing of multiple log records to be pushed is performed according to the push conditions corresponding to the configuration information, it also includes: in response to the failure of pushing the multiple log records to be pushed, determining the number of failures corresponding to the multiple log records to be pushed; in response to the number of failures being less than a number threshold, re-pushing the multiple log records to be pushed; in response to the number of failures being not less than the number threshold, generating alarm information for the multiple log records to be pushed.

[0065] If the number of failures is less than the number threshold, after a preset period of time, the multiple log records to be pushed are pushed again. If the number of failures is not less than the number threshold, an alarm message is generated for the multiple log records to be pushed, so that relevant personnel can quickly find out the cause of the log push failure.

[0066] In one embodiment of the present invention, obtaining at least one piece of configuration information corresponding to the scene information includes: in response to the scene information indicating that no abnormal event occurs in the current scene, determining the configuration information corresponding to the normal event type as the configuration information corresponding to the scene information.

[0067] In the case where no abnormal warning information for the target system is received, or no abnormal operation of the target system is detected, or the user does not start abnormal event analysis for the target system, etc., it can be determined that the scene information represents that no abnormal event has occurred in the current scene, and the configuration information corresponding to the normal event type is determined as the configuration information corresponding to the scene information. According to the configuration information corresponding to the normal event type, the log record to be pushed corresponding to the normal event type is obtained, and the log record is collected and pushed. Through the pushed log record, the target system is fully monitored.

[0068] Specifically, based on the log records to be pushed corresponding to the normal event types, daily business operations analysis is performed to formulate subsequent operation strategies. Furthermore, the log records to be pushed corresponding to the normal event types can also be used to discover hidden abnormal situations that have not yet been discovered. For example, based on the log records to be pushed corresponding to the normal event types, statistical indicators such as page views, number of failed requests, and request time can be calculated. Based on the obtained statistical indicators, monitor whether other abnormal events have occurred. It is also possible to track links based on the log records to be pushed corresponding to the normal event types to determine whether users access the website through normal links, etc.

[0069] Figure 2 FIG. 1 is a schematic diagram of a log push method process provided by another embodiment of the present invention. Figure 2 As shown, the method includes:

[0070] Step 201: In response to scene information indicating that an abnormal event occurs in the current scene, at least one event type corresponding to the abnormal event is determined.

[0071] The event type can be set according to business needs. At least one event type corresponding to the abnormal event can be determined based on the abnormal warning information of the target system received or monitored, or the abnormal event analysis information of the target system that the user has not started, etc. The event type can be: BOT attack, DDoS attack, overseas attack, page abnormality, etc.

[0072] Step 202: Determine configuration information corresponding to the scene information based on configuration information corresponding to at least one event type.

[0073] Configuration information includes: multiple data fields, verification rules, sampling ratios, and push conditions. An event type can correspond to at least one piece of configuration information. The verification rules for event types can be set according to the characteristics of the event type and business requirements.

[0074] The configuration information corresponding to each event type corresponding to the scene information can be used to determine the configuration information corresponding to the scene information. According to the configuration information corresponding to each event type, the log records to be pushed corresponding to each event type are determined. Based on the log records to be pushed corresponding to the event type, the abnormal events corresponding to the event type are analyzed and processed.

[0075] In one embodiment of the present invention, configuration information corresponding to scene information is determined based on configuration information corresponding to at least one event type, including: determining configuration information corresponding to each event type; determining configuration information corresponding to a normal event type; and determining the configuration information corresponding to each event type and the configuration information corresponding to the normal event type as configuration information corresponding to the scene information.

[0076] The configuration information corresponding to each event type and the configuration information corresponding to the normal event type are determined as the configuration information corresponding to the scene information. Based on the log records to be pushed corresponding to the event type, the abnormal events corresponding to the event type are analyzed and processed.

[0077] At the same time, according to the configuration information corresponding to the normal event type, the log records to be pushed corresponding to the normal event type are determined, and based on the log records to be pushed corresponding to the normal event type, daily business operation analysis is performed to formulate subsequent operation strategies. Furthermore, the log records to be pushed corresponding to the normal event type can also be used to discover hidden abnormal situations that have not yet been discovered. For example, based on the log records to be pushed corresponding to the normal event type, statistical indicators such as page views, number of failed requests, and request time can be calculated. Based on the obtained statistical indicators, monitor whether other abnormal events have occurred. It is also possible to track links based on the log records to be pushed corresponding to the normal event type to determine whether the user accesses the website through a normal link. By analyzing the log records to be pushed corresponding to the normal event type, the normal operation of the target system can be ensured.

[0078] Step 203: Obtain multiple log records within the statistical period.

[0079] Step 204: for each piece of configuration information, generate a filtering log record corresponding to each log record according to multiple data fields corresponding to the configuration information; and filter the multiple filtering log records according to the verification rules corresponding to the configuration information to obtain multiple filtering log records.

[0080] Step 205: Determine multiple log records to be pushed from multiple filtered log records according to the sampling ratio corresponding to the configuration information; and perform push processing on the multiple log records to be pushed according to the push conditions corresponding to the configuration information.

[0081] In the scheme of the embodiment of the present invention, with respect to the situation where the scene information represents an abnormal event occurring in the current scene, the configuration information corresponding to the scene information is determined according to at least one event type corresponding to the abnormal event, and then the log records to be pushed corresponding to each event type are determined, and based on the log records to be pushed corresponding to the event type, the abnormal events corresponding to the event types are analyzed and processed, which can reduce or avoid the adverse effects of the abnormal events on the system.

[0082] Figure 3 FIG. 1 is a schematic diagram of a process of a log push method provided by another embodiment of the present invention. Figure 3 As shown, the method includes:

[0083] Step 301: Obtain at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios, push conditions, output formats, and storage servers.

[0084] Step 302: Obtain multiple log records within the statistical period.

[0085] Step 303: for each piece of configuration information, generate a filtering log record corresponding to each log record according to multiple data fields corresponding to the configuration information; and filter the multiple filtering log records according to the verification rules corresponding to the configuration information to obtain multiple filtering log records.

[0086] Step 304: Determine a plurality of log records to be pushed from the plurality of filtered log records according to the sampling ratio corresponding to the configuration information.

[0087] Step 305: convert the format of the multiple log records to be pushed according to the output format.

[0088] The output format can be set as needed, and the output format can be ndjson, csv, json, etc. The format of multiple log records to be pushed is converted to convert multiple log records to be pushed into log records in the output format.

[0089] Step 306: according to the push conditions corresponding to the configuration information, push processing is performed on the converted multiple log records to be pushed, so as to push the converted multiple log records to be pushed to the storage server.

[0090] Multiple storage servers may be set up, and different storage servers are used to analyze and monitor log records to be pushed corresponding to different event types, or to analyze and monitor log records to be pushed corresponding to different application systems, etc.

[0091] In the solution of the embodiment of the present invention, the configuration information also includes: output format and storage server. According to the output format and storage server, the storage format and push service of the log record are set to perform accurate analysis and monitoring of the log records of each event type or application system.

[0092] Figure 4 FIG. 1 is a schematic diagram of the structure of a log push system provided by an embodiment of the present invention. Figure 4 As shown in the figure, the system consists of two parts: the console and the data plane. The console is used to configure the relevant information of the log push task, including the configuration data set, data field matching related policies, and storage servers, and to enable abnormal log analysis. The data field matching related policies can include configuration information corresponding to multiple event types.

[0093] After the configuration is completed, the data plane will start the log push job according to the relevant information of the configured log push task, sample the log records according to the matching data set, multiple data fields, verification rules, sampling ratio, push conditions, output format and storage server in the configuration information, and perform attack analysis. Finally, by establishing a connection with the storage server, the log records of the statistical period are pushed to the storage server in batches. Finally, you can view the detailed push records of the log records in the console, and make manual supplementary pushes or monitoring alarms for sudden abnormal situations. Users can configure the relevant information of the log push task according to the following steps:

[0094] Step S11: Select the data set to collect log records, such as Figure 4 HTTP request logs, firewall event logs, etc.

[0095] Step S12: Determine multiple data fields corresponding to the configuration information.

[0096] Step S13: Select the log output format corresponding to the configuration information, such as ndjson, csv, json, etc.

[0097] Step S14: Select the sampling ratio corresponding to the configuration information, and the system will sample and push logs according to the configured ratio.

[0098] Step S15: Select push strategy: Enable attack event analysis, select the event type to be analyzed, the system analyzes the attack behavior according to the event type, and pushes the result set. The event type of the attack event may include: BOT attack, etc. For BOT attacks, the behavior will be analyzed according to the BOT score to determine whether the request is a normal request from the user or a request from an automatic program.

[0099] Step S16: Set the verification rules corresponding to the configuration information. The system only selects the log records that pass the verification rules for push.

[0100] Step S17: Select the storage server corresponding to the configuration information. You can configure the URL (Uniform Resource Locator) corresponding to the storage server, the authorization token, whether to skip security verification, etc.

[0101] Figure 5 FIG. 1 is a schematic diagram of a log push method provided by another embodiment of the present invention. Figure 5 As shown in the figure, after the console configures the relevant information of the log push task, the task is started and the data plane starts the log push job. The system obtains the task policy configured in the console and obtains log records in batches. The process of pushing logs may include the following steps:

[0102] Step S21: The received log has all fields. According to the multiple data fields in the configuration information, the data fields that do not need to be pushed are filtered out. And through the pre-filter, the log records that do not match the verification rules are recursively filtered out. The verification rules provide solutions for complex scenarios of enterprises, which facilitates direct analysis and prediction after obtaining the log records.

[0103] Step S22: Sample a batch of log records through the reservoir algorithm to obtain log records to be pushed. If the total amount of data of the collected log records is s and the sampling ratio is q, the number of log records to be pushed this time is k=s*q. The system will first store the first k logs in the queue to be pushed, and start selecting from the k+1th log data with a probability of k / m, and replace a log data in the queue to be pushed with a probability of 1 / k, so as to sample the log records with equal probability.

[0104] Step S23: After sampling, each batch of logs will be stored in the queue to be pushed. When the queue meets the push conditions, a connection is created to push the log records in the queue to be pushed to the storage server in the log format. The push conditions can be a preset time point, a preset number, etc. For example, if the number of log records stored in the queue to be pushed exceeds 1,000, or if no log records are pushed for more than 1 minute, log push processing will be performed.

[0105] Step S24: If the user starts attack event analysis for an event type, the system will perform behavioral analysis and source tracing of the attack by pushing log records to the storage service when an attack occurs. By analyzing the log records corresponding to the event type, the attack intent can be clarified, which facilitates the enterprise to quickly locate the attack, as well as the subsequent report sorting, analysis and review.

[0106] For events of BOT attack, the BOT score is used as a benchmark to determine the probability of the request being a machine-automated request or a normal user request. At the same time, based on BOT automatic program analysis, BOT automatic program score source, BOT request score distribution data, etc., the relevant data of the attack behavior is aggregated according to the preset time.

[0107] For DDoS analysis, logs will be aggregated and counted during the attack, including DDoS attack traffic and bandwidth.

[0108] For events of overseas attacks, since overseas attacks account for a high proportion in network security, in log analysis, the IP, domain name, etc. are traced through the filing system, the overseas address tags are aggregated, and relevant traffic data is pushed.

[0109] For event types such as page exceptions, when a website log receives a large number of abnormal status codes such as 404, or the number of abnormal status codes exceeds the threshold, an alarm log will be triggered and pushed to the storage server. The alarm log may include: corresponding URL, status code, proxy, and frequency.

[0110] Attacks of all types will trigger attack alerts that generate abnormal events. Attack alerts can be notified to enterprises for prevention and disposal via email, phone, text messages, etc.

[0111] The solution of the embodiment of the present invention also provides a fallback strategy. During the push process of log records, if the connection is not smooth due to network or other reasons, the push will fail, and the fallback strategy will be executed at this time. The fallback strategy is divided into two scenarios. In the first scenario, when the push fails, it will be re-pushed a preset number of times. If the push fails, an email notification will be sent for manual intervention. In the second scenario, a manual re-push mechanism is provided. By selecting a certain time period to trigger the re-push, the log push system will find the full log of this interval in batches, and execute the above steps S21-S24 to re-push and save the full log of this interval. During the log push process, the push results of the log records are saved, and users can view the log push records on the console.

[0112] The solution of the embodiment of the present invention is suitable for log push of various security business scenarios, and can push massive attack logs in real time. The solution of the embodiment of the present invention has intelligent log filtering and dynamic collection. Users can configure various attack behavior analysis and anomaly detection and other related strategies to meet various security scenarios of the enterprise. It also provides a flexible alarm mechanism and a bottom-up retry strategy to ensure that log records can be pushed smoothly.

[0113] Figure 6 FIG. 1 is a schematic diagram of a log push device provided by an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0114] The information acquisition module 601 is used to acquire at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions;

[0115] The record acquisition module 602 is used to acquire multiple log records within the statistical period;

[0116] The log push module 603 is used to generate, for each configuration information, a filtering log record corresponding to each log record according to multiple data fields corresponding to the configuration information; filter multiple filtering log records according to the verification rules corresponding to the configuration information to obtain multiple filtering log records; determine multiple log records to be pushed from the multiple filtering log records according to the sampling ratio corresponding to the configuration information; and push the multiple log records to be pushed according to the push conditions corresponding to the configuration information.

[0117] Optionally, the information acquisition module 601 is specifically used for:

[0118] In response to the scene information indicating that an abnormal event occurs in the current scene, determining at least one event type corresponding to the abnormal event;

[0119] Configuration information corresponding to the scene information is determined according to configuration information corresponding to at least one event type.

[0120] Optionally, the information acquisition module 601 is specifically used for:

[0121] Determine the configuration information corresponding to each event type;

[0122] Determine the configuration information corresponding to the normal event type;

[0123] The configuration information corresponding to each event type and the configuration information corresponding to the normal event type are determined as the configuration information corresponding to the scene information.

[0124] Optionally, the information acquisition module 601 is specifically used for:

[0125] In response to the scene information indicating that no abnormal event occurs in the current scene, the configuration information corresponding to the normal event type is determined as the configuration information corresponding to the scene information.

[0126] Optionally, the configuration information also includes: output format and storage server;

[0127] The log push module 603 is specifically used for:

[0128] According to the output format, multiple log records to be pushed are converted into different formats;

[0129] According to the push conditions corresponding to the configuration information, push processing is performed on the converted multiple log records to be pushed, so as to push the converted multiple log records to be pushed to the storage server.

[0130] Optionally, the log push module 503 is further used for:

[0131] In response to failure in pushing the plurality of log records to be pushed, determining the number of failures corresponding to the plurality of log records to be pushed;

[0132] In response to the number of failures being less than the number threshold, re-pushing the plurality of log records to be pushed;

[0133] In response to the number of failures being not less than a number threshold, generating alarm information for the plurality of log records to be pushed.

[0134] An embodiment of the present invention provides an electronic device, including:

[0135] one or more processors;

[0136] The storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0137] An embodiment of the present invention provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor.

[0138] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0139] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0140] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0141] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0142] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, they may be described as: an information acquisition module, a record acquisition module, and a log push module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the information acquisition module may also be described as a "module for acquiring at least one piece of configuration information corresponding to the scene information."

[0145] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0146] Obtain at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios, and push conditions;

[0147] Get multiple log records within the statistical period;

[0148] For each piece of configuration information, according to the multiple data fields corresponding to the configuration information, a filtering log record corresponding to each log record is generated; according to the verification rules corresponding to the configuration information, multiple filtering log records are filtered to obtain multiple filtering log records; according to the sampling ratio corresponding to the configuration information, multiple log records to be pushed are determined from the multiple filtering log records; according to the push conditions corresponding to the configuration information, the multiple log records to be pushed are pushed.

[0149] According to the technical solution of an embodiment of the present invention, at least one configuration information corresponding to the scene information is selected, and the log record is processed and pushed according to the configuration information. The configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions. During the log record processing, it is only necessary to retain the values ​​of each data field corresponding to the configuration information, and select the log record according to the verification rules and the adoption ratio. Compared with the method of pushing the full amount of logs, it can reduce the pressure on the log analysis server.

[0150] In addition, different configuration information is set according to different scenarios of the system. The pushed log records are more in line with business needs, and the information in the log records is more targeted, which is convenient for subsequent log analysis and attack prediction.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A log push method, characterized in that: include: Obtain at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios, and push conditions; Get multiple log records within the statistical period; For each of the configuration information, based on the multiple data fields corresponding to the configuration information, a filtering log record corresponding to each of the log records is generated; based on the verification rules corresponding to the configuration information, multiple filtering log records are filtered to obtain multiple filtering log records; based on the sampling ratio corresponding to the configuration information, multiple log records to be pushed are determined from the multiple filtering log records; based on the push conditions corresponding to the configuration information, the multiple log records to be pushed are pushed.

2. The method according to claim 1, characterized in that: The obtaining of at least one piece of configuration information corresponding to the scene information includes: In response to the scene information representing that an abnormal event occurs in the current scene, determining at least one event type corresponding to the abnormal event; According to the configuration information corresponding to the at least one event type, the configuration information corresponding to the scene information is determined.

3. The method according to claim 2, characterized in that The determining, according to the configuration information corresponding to the at least one event type, the configuration information corresponding to the scene information includes: Determine configuration information corresponding to each of the event types; Determine the configuration information corresponding to the normal event type; The configuration information corresponding to each of the event types and the configuration information corresponding to the normal event type are determined as the configuration information corresponding to the scene information.

4. The method according to claim 1, characterized in that: The obtaining of at least one piece of configuration information corresponding to the scene information includes: In response to the scene information indicating that no abnormal event occurs in the current scene, the configuration information corresponding to the normal event type is determined as the configuration information corresponding to the scene information.

5. The method according to claim 1, characterized in that: The configuration information also includes: output format and storage server; The pushing process of the plurality of log records to be pushed according to the push conditions corresponding to the configuration information includes: According to the output format, converting the plurality of log records to be pushed; According to the push conditions corresponding to the configuration information, push processing is performed on the converted multiple log records to be pushed, so as to push the converted multiple log records to be pushed to the storage server.

6. The method according to claim 1, characterized in that After the plurality of log records to be pushed are pushed according to the push conditions corresponding to the configuration information, the method further includes: In response to failure in pushing the plurality of log records to be pushed, determining the number of failures corresponding to the plurality of log records to be pushed; In response to the number of failures being less than a number threshold, re-pushing the plurality of log records to be pushed; In response to the number of failures being not less than a number threshold, generating alarm information for the plurality of log records to be pushed.

7. A log pushing device, characterized in that: include: An information acquisition module, used to acquire at least one piece of configuration information corresponding to the scene information; wherein the configuration information includes: multiple data fields, verification rules, sampling ratios and push conditions; The record acquisition module is used to obtain multiple log records within the statistical period; The log push module is used to generate, for each of the configuration information, a filtering log record corresponding to each of the log records according to a plurality of data fields corresponding to the configuration information; filter a plurality of the filtering log records according to a verification rule corresponding to the configuration information to obtain a plurality of filtering log records; determine a plurality of log records to be pushed from the plurality of filtering log records according to a sampling ratio corresponding to the configuration information; and push the plurality of log records to be pushed according to a push condition corresponding to the configuration information.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images