Multi-application-oriented access log anomaly detection method and program product

Abstract

The invention discloses a multi-application-oriented access log anomaly detection method and a program product. The method comprises the steps that log collection is carried out on corresponding application nodes through application node log collectors in a log collection module, and target access logs to be subjected to anomaly detection are obtained; processing each target access log to be subjected to exception detection through a log preprocessing module according to a log preprocessing method to obtain each standard access log field to be subjected to exception detection; analyzing and storing the log through log index configuration and storage rules; and obtaining a received application access log detection task, and according to the received application access log detection task, performing anomaly detection on each stored standard storage access log field to be subjected to anomaly detection through an abnormal behavior detection module to generate an access log anomaly detection result. The problems that manual analysis is low in efficiency and accuracy, and log analysis cannot be carried out in a centralized mode are solved, and the accuracy and efficiency of access log anomaly detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network security data processing, and in particular to a multi-application-oriented access log anomaly detection method and program product. BACKGROUND

[0002] Traditional Internet application access logs are generally stored in local files, and the analysis method is mainly periodic export, which cannot achieve real-time and intelligent anomaly identification. Specifically, for financial Internet related applications, abnormal transaction logs and application access are particularly sensitive. In the current severe security situation, real-time log monitoring is required.

[0003] The inventor found that the prior art has the following defects in the process of implementing the present application: At present, there are problems of data dispersion and poor real-time performance, and the logs of multiple application nodes cannot be centrally analyzed; unstructured logs cannot be effectively processed, and there is a lack of automatic alarm mechanism; most operation and maintenance personnel perform manual analysis through command line tools, which is low in efficiency and accuracy. SUMMARY

[0004] The present application provides a multi-application-oriented access log anomaly detection method and program product to improve the accuracy and efficiency of access log anomaly detection.

[0005] According to an aspect of the present application, a multi-application-oriented access log anomaly detection method is provided, which comprises:

[0006] The application node log collector in the log collection module collects access logs of the corresponding application node respectively to obtain each target access log to be abnormally detected, and sends each target access log to be abnormally detected to the log preprocessing module through the log collection agent plug-in;

[0007] The multi-application-oriented access log anomaly detection system comprises a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module; the multi-application-oriented access log anomaly detection system is used for detecting access logs of multiple applications, each application corresponds to an application node, and an application node log collector is deployed on the application node;

[0008] The log preprocessing module processes each target access log to be abnormally detected according to a pre-set log preprocessing method to obtain each standard access log field to be abnormally detected;

[0009] The log indexing and storage module analyzes and stores each standard access log field to be abnormally detected according to the log indexing and storage rules to obtain each standard stored access log field to be abnormally detected;

[0010] Based on the received application access log detection task, the abnormal behavior detection module performs anomaly detection on the fields of the standard storage access logs to be detected, generates the corresponding access log anomaly detection results for each application, and feeds them back.

[0011] According to another aspect of the present invention, an access log anomaly detection device for multiple applications is provided, comprising:

[0012] The target anomaly detection access log sending module is used to collect access logs of the corresponding application nodes through the log collectors of each application node in the log collection module, obtain the target anomaly detection access logs, and send the target anomaly detection access logs to the log preprocessing module through the log collection proxy plugin.

[0013] The multi-application access log anomaly detection system includes a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module. The multi-application access log anomaly detection system is used to detect the access logs of multiple applications. Each application corresponds to an application node, and an application node log collector is deployed on the application node.

[0014] The standard access log field determination module is used to process each target access log to be detected according to a pre-set log preprocessing method through the log preprocessing module to obtain each standard access log field to be detected.

[0015] The standard storage access log field determination module is used to analyze and store each of the standard access log fields to be detected by anomaly detection through the log index and the log index configuration and storage rules in the storage module, so as to obtain the stored standard storage access log fields to be detected by anomaly detection.

[0016] The access log anomaly detection result generation and feedback module is used to acquire and, based on the received application access log detection task, perform anomaly detection on the fields of the access logs to be detected in each standard storage through the anomaly behavior detection module, generate access log anomaly detection results corresponding to each application, and provide feedback.

[0017] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-application access log anomaly detection method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the multi-application access log anomaly detection method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the multi-application access log anomaly detection method described in any embodiment of the present invention.

[0020] The technical solution of this invention involves collecting access logs from each application node in the log collection module to obtain target access logs to be detected for anomalies. These target access logs are then sent to the log preprocessing module via a log collection proxy plugin. The log preprocessing module processes these target access logs according to a pre-set log preprocessing method to obtain standard access log fields to be detected for anomalies. The standard access log fields to be detected for anomalies are analyzed and stored using log index configuration and storage rules in the log indexing and storage module to obtain stored standard access log fields to be detected for anomalies. Finally, based on the received application access log detection task, the abnormal behavior detection module performs anomaly detection on these standard stored access log fields to be detected for anomalies, generating and feeding back access log anomaly detection results for each application. It solves the problems of low efficiency and accuracy caused by manual analysis, inability to centralize log analysis, lack of automatic alarm mechanism, and ineffective processing of unstructured logs. It enables centralized analysis and processing of logs corresponding to different applications, can perform automatic alarm operations, and effectively analyze and process unstructured logs, thereby improving the accuracy and efficiency of access log anomaly detection.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a flowchart of an access log anomaly detection method for multiple applications provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of an access log anomaly detection device for multiple applications provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of an electronic device provided according to Embodiment 3 of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "target," "current," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] It is worth noting that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse; if the user chooses to refuse, the process will proceed to the expert decision-making process.

[0029] Example 1

[0030] Figure 1The flowchart of the method for detecting access log anomalies for multiple applications is provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of detecting anomalies in the access logs of multiple applications. The method can be executed by a device for detecting access log anomalies for multiple applications, which can be implemented in hardware and / or software.

[0031] Correspondingly, such as Figure 1 As shown, the method includes:

[0032] S110. The access logs of the corresponding application nodes are collected by the log collectors of each application node in the log collection module to obtain the access logs of each target to be detected for anomalies, and the access logs of each target to be detected for anomalies are sent to the log preprocessing module through the log collection proxy plugin.

[0033] The multi-application access log anomaly detection system includes a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module. The multi-application access log anomaly detection system is used to detect the access logs of multiple applications. Each application corresponds to an application node, and an application node log collector is deployed on the application node.

[0034] In this embodiment, access logs can be collected from the unstructured device log files in the distributed server corresponding to each application node using a log collector for each application node. The unstructured device can be a lightweight web server, a reverse proxy server, an email proxy server, or a device where application deployment logs are stored.

[0035] The log collection agent plugin is used to collect logs from various systems or applications, convert them into a user-specified format, and then forward them to the user-specified log storage system. It can also send access logs for anomaly detection from the log collection module to the log preprocessing module.

[0036] S120. The log preprocessing module processes the access logs of each target to be detected as an anomaly according to the pre-set log preprocessing method to obtain the access log fields of each standard to be detected as an anomaly.

[0037] The log preprocessing method may include methods that include field parsing and format structuring.

[0038] Optionally, the step of processing the target access logs to be detected for anomaly detection using the log preprocessing module according to a pre-set log preprocessing method to obtain standard access log fields to be detected for anomaly detection includes: parsing the fields of each target access log to be detected for anomaly detection using the field parsing sub-method corresponding to the log preprocessing method, and filtering invalid fields after parsing to obtain parsed fields for each target access log; and performing structured conversion processing on the parsed fields of each target access log using the format structuring sub-method corresponding to the log preprocessing method, and filtering abnormal formats after conversion to obtain standard access log fields to be detected for anomaly detection corresponding to each target access log to be detected for anomaly detection; wherein, the standard access log fields to be detected for anomaly detection include a remote address field, a log request field, a log status code field, a user source field, a time field, an Internet Protocol address field, and an access path field.

[0039] The log status code field can be used to describe whether a correct or incorrect status code was returned. The user source field can contain information about the device accessed by the user.

[0040] In this embodiment, unstructured access logs need to be processed by a log preprocessing module to generate standard structured logs. Specifically, the access logs of each target to be detected for anomalies first need to be parsed, resulting in multiple parsed fields. A pre-set standard field matching library is then obtained, and the parsed fields are matched against the standard field matching library. Fields that match are retained, while fields that do not match are filtered out as invalid fields, thus obtaining the parsed fields for each target access log.

[0041] Furthermore, a structured processing sub-method is needed to perform structured transformation on the parsed fields of each target access log, and to filter out abnormal formats in the transformed fields. Specifically, a standard format matching library can be set up, and the data can be compared with the standard format matching library. Formats that cannot be matched are considered abnormal formats, and these abnormal formats are filtered out to obtain the standard access log fields to be detected for anomalies.

[0042] The advantage of this setup is that by parsing the fields and filtering invalid fields, as well as performing structure conversion and abnormal format filtering on the target access logs to be detected, the resulting standard access log fields to be detected can effectively process unstructured logs, improving the flexibility and diversity of log processing.

[0043] S130. By configuring the log index and storage rules in the log index and storage module, the standard access log fields to be detected for anomaly detection are analyzed and stored to obtain the stored standard access log fields to be detected for anomaly detection.

[0044] The log index configuration and storage rules can be based on multiple fields to index and determine the target storage location for accessing log storage rules.

[0045] Optionally, the step of analyzing and storing the standard access log fields to be detected for anomaly detection through the log index configuration and storage rules in the log indexing and storage module to obtain the stored standard access log fields to be detected for anomaly detection includes: according to the log index configuration and storage rules, using the target distributed search and analysis engine to perform index matching analysis on the time field, Internet protocol address field, access path field, and log status code field in the standard access log fields to be detected for anomaly detection; if the index matching analysis result meets the log field storage conditions, then the remote address field, log request field, log status code field, user source field, time field, Internet protocol address field, and access path field are stored in the target log field repository to obtain the stored standard access log fields to be detected for anomaly detection; wherein, different applications correspond to different Internet protocol address fields, and the different field types corresponding to different applications are stored in different locations in the target log field repository; if the index matching analysis result does not meet the log field storage conditions, then a storage error instruction is fed back to the user.

[0046] Among them, the target distributed search and analysis engine is a retrieval platform that can store structured, unstructured and vector data in real time, provide fast hybrid and vector search, support observability and security analysis, and enable artificial intelligence-driven applications with high performance, high accuracy and high relevance.

[0047] In this embodiment, the time field, Internet Protocol address field, access path field, and log status code field are first analyzed for index matching to determine whether there is a target index storage location. If there is, it can be determined that the index matching analysis result meets the log field storage conditions. Then, the remote address field, log request field, log status code field, user source field, time field, Internet Protocol address field, and access path field corresponding to the standard access log field to be detected for anomaly are stored. And the next standard access log field to be detected for anomaly is analyzed and processed.

[0048] Conversely, if the target's index storage location does not exist, feedback needs to be provided to the user, indicating a storage error instruction. The target's storage location needs to be determined through manual analysis, and then the new storage location can be added to the index storage location library for update and optimization.

[0049] The advantage of this setup is that by configuring log indexes and storage rules, the access log fields for each standard anomaly detection can be analyzed and stored. This allows for effective and reasonable log storage, centralized storage and analysis of logs from multiple application nodes, and significantly improves log processing efficiency.

[0050] Optionally, the multi-application access log anomaly detection system further includes a log aggregation and summarization module; after analyzing and storing each of the standard access log fields to be detected through the log index configuration and storage rules in the log index and storage module to obtain the stored standard access log fields to be detected, the system further includes: receiving a target log aggregation and summarization task in real time through the log aggregation and summarization module, and parsing the target log aggregation and summarization task to obtain a target log aggregation and summarization parsing result; wherein the target log aggregation and summarization parsing result includes a target Internet Protocol address field and a target time field; and summarizing the corresponding standard access log fields to be detected based on the target Internet Protocol address field and the target time field to obtain a standard summary statistical result; wherein the standard summary statistical result includes at least one of the following: a statistical result of the number of accesses per minute for different applications, a statistical result of the frequency ranking of the access Internet Protocol address field, a statistical result of the status code error rate, and a statistical result of the aggregated distribution of the access path field.

[0051] In this embodiment, a target log aggregation task can be received in real time and parsed to obtain the specific target Internet Protocol address field and target time field. For example, parsing the received target log aggregation task A reveals that the application corresponding to the target Internet Protocol address field is application 11, and the target time field is from 8:00 to 9:00 on November 1, 2025.

[0052] Therefore, based on the above analysis results, it is necessary to summarize the standard storage access log fields for anomaly detection in Application 11 from 8:00 AM to 9:00 AM on November 1, 2025, to obtain the summary statistical results for each standard. This could include detailed statistical results such as the number of accesses per minute for the target application during the target time period, the frequency ranking of access to the Internet Protocol address field, the status code error rate, and the aggregated distribution of the access path field.

[0053] The advantage of this setup is that by aggregating and summarizing the received target logs, the access log fields of each standard storage to be detected for anomalies are summarized and processed. This allows for better display and processing of the access log fields of each standard storage to be detected for anomalies, reflects the log access status of different applications in real time, and enables centralized analysis of logs from different applications at different points in time.

[0054] S140. Obtain and, based on the received application access log detection task, perform anomaly detection on the fields of the standard stored access logs to be detected through the anomaly detection module, generate the access log anomaly detection results corresponding to each application, and provide feedback.

[0055] Among them, the application access log detection task can be a task that performs anomaly detection on the access log fields to be detected in each standard storage, or on the standard summary statistical results.

[0056] Optionally, the step of acquiring and, based on the received application access log detection task, performing anomaly detection on each standard storage access log field to be detected through the anomaly behavior detection module, generating access log anomaly detection results corresponding to each application and feeding them back includes: parsing the received application access log detection task to obtain the target application to be detected and the target time to be detected; filtering the target standard storage access log fields to be detected that correspond to the target application to be detected and the target time to be detected from each standard storage access log field to be detected; using the anomaly behavior detection module to perform anomaly detection on the target standard storage access log fields to be detected, generating access log anomaly detection results corresponding to the target application and feeding them back.

[0057] In this embodiment, based on the target application to be detected and the target time to be detected corresponding to the application access log detection task, it is necessary to filter out the corresponding target standard storage access log fields to be detected for anomaly detection, so as to perform anomaly detection and generate the access log anomaly detection result corresponding to the target application.

[0058] Specifically, anomaly detection can include frequency anomaly detection. Specifically, it is based on the same Internet Protocol address requests within a time window (such as 1 minute or 5 minutes) to determine whether the number of requests exceeds a preset threshold. If it does, it can be identified as an anomaly.

[0059] Furthermore, anomaly detection can also include behavioral pattern anomaly detection. Specifically, it can be based on the characteristics of the access path field (such as probe path or interface abuse) to determine behavioral anomalies; it can also be based on the distribution of status codes to determine whether the status codes are correct or incorrect, thereby determining whether behavioral anomalies exist.

[0060] Correspondingly, anomaly detection can also include a combination of template matching and semi-supervised learning. Specifically, templates can be constructed for typical attack behaviors (such as scanning or structured query language injection attempts), and semi-supervised learning methods can be used to achieve dynamic anomaly modeling.

[0061] The advantages of this setup are: it enables the analysis of abnormal access logs of different applications at different time periods based on real-time received application access log detection tasks, improving the flexibility, efficiency, and accuracy of access log anomaly analysis, reducing manual costs and workload, and enhancing user experience.

[0062] Optionally, after the abnormal behavior detection module performs anomaly detection on the target standard storage access log fields to be detected, the method further includes: if there is anomaly in the access log fields, the target alarm mechanism is invoked through the alarm and linkage module in the multi-application access log anomaly detection system; the target automatic linkage defense strategy corresponding to the target alarm mechanism is obtained, so as to realize access log anomaly alarm processing according to the target alarm mechanism, and to perform automated processing of abnormal access logs in combination with the target automatic linkage defense strategy.

[0063] In this embodiment, abnormal access log fields can be automatically alerted. This allows for the acquisition of the target automatic linkage defense strategy corresponding to the target alert mechanism. Prior to this, historical abnormal access log fields, along with corresponding historical alert methods and automatic linkage defense strategies, can be jointly stored. After determining the abnormal access log fields, the target alert method and target automatic linkage defense strategy can be determined accordingly.

[0064] Furthermore, alerts can be sent to users via targeted methods, such as email or SMS, and to operations and maintenance support personnel, and automatic linkage defense strategies can be implemented to ensure the network security of the application.

[0065] The advantage of this setup is that by handling abnormal access log alerts based on the target alert mechanism and combining them with the target's automatic linkage defense strategy to automate the processing of abnormal access logs, abnormal access logs can be detected and processed in a timely manner, ensuring the network security of the application and improving the efficiency and accuracy of abnormal access log detection.

[0066] Additionally, the management side can view and manage the overall log status and alarm history through a visual platform.

[0067] The technical solution of this invention involves collecting access logs from each application node in the log collection module to obtain target access logs to be detected for anomalies. These target access logs are then sent to the log preprocessing module via a log collection proxy plugin. The log preprocessing module processes these target access logs according to a pre-set log preprocessing method to obtain standard access log fields to be detected for anomalies. The standard access log fields to be detected for anomalies are analyzed and stored using log index configuration and storage rules in the log indexing and storage module to obtain stored standard access log fields to be detected for anomalies. Finally, based on the received application access log detection task, the abnormal behavior detection module performs anomaly detection on these standard stored access log fields to be detected for anomalies, generating and feeding back access log anomaly detection results for each application. It solves the problems of low efficiency and accuracy caused by manual analysis, inability to centralize log analysis, lack of automatic alarm mechanism, and ineffective processing of unstructured logs. It enables centralized analysis and processing of logs corresponding to different applications, can perform automatic alarm operations, and effectively analyze and process unstructured logs, thereby improving the accuracy and efficiency of access log anomaly detection.

[0068] Example 2

[0069] Figure 2 This is a schematic diagram of a multi-application access log anomaly detection device provided in Embodiment 2 of the present invention. The multi-application access log anomaly detection device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or server to implement a multi-application access log anomaly detection method according to the present invention. Figure 2 As shown, the device includes: a target anomaly detection access log sending module 210, a standard anomaly detection access log field determination module 220, a standard storage anomaly detection access log field determination module 230, and an access log anomaly detection result generation and feedback module 240.

[0070] The target anomaly detection access log sending module 210 is used to collect access logs of the corresponding application nodes through the application node log collectors in the log collection module, obtain the target anomaly detection access logs, and send the target anomaly detection access logs to the log preprocessing module through the log collection proxy plugin.

[0071] The multi-application access log anomaly detection system includes a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module. The multi-application access log anomaly detection system is used to detect the access logs of multiple applications. Each application corresponds to an application node, and an application node log collector is deployed on the application node.

[0072] The standard access log field determination module 220 is used to process each target access log to be detected by an anomaly detection through the log preprocessing module according to a pre-set log preprocessing method to obtain each standard access log field to be detected by anomaly detection.

[0073] The standard storage access log field determination module 230 is used to analyze and store each of the standard access log fields to be detected by anomaly detection through the log index and the log index configuration and storage rules in the storage module, so as to obtain the stored standard storage access log fields to be detected by anomaly detection.

[0074] The access log anomaly detection result generation and feedback module 240 is used to acquire and, based on the received application access log detection task, perform anomaly detection on the fields of the access logs to be detected in each standard storage through the anomaly behavior detection module, generate access log anomaly detection results corresponding to each application, and provide feedback.

[0075] The technical solution of this invention involves collecting access logs from each application node in the log collection module to obtain target access logs to be detected for anomalies. These target access logs are then sent to the log preprocessing module via a log collection proxy plugin. The log preprocessing module processes these target access logs according to a pre-set log preprocessing method to obtain standard access log fields to be detected for anomalies. The standard access log fields to be detected for anomalies are analyzed and stored using log index configuration and storage rules in the log indexing and storage module to obtain stored standard access log fields to be detected for anomalies. Finally, based on the received application access log detection task, the abnormal behavior detection module performs anomaly detection on these standard stored access log fields to be detected for anomalies, generating and feeding back access log anomaly detection results for each application. It solves the problems of low efficiency and accuracy caused by manual analysis, inability to centralize log analysis, lack of automatic alarm mechanism, and ineffective processing of unstructured logs. It enables centralized analysis and processing of logs corresponding to different applications, can perform automatic alarm operations, and effectively analyze and process unstructured logs, thereby improving the accuracy and efficiency of access log anomaly detection.

[0076] Based on the above embodiments, the standard access log field determination module 220 can be specifically used to: perform field parsing processing on each of the target access logs to be detected by the field parsing processing sub-method corresponding to the log preprocessing method, and perform invalid field filtering processing on the parsed fields to obtain parsed fields for each target access log; perform structure conversion processing on each of the target access log parsing fields by the format structuring processing sub-method corresponding to the log preprocessing method, and perform abnormal format filtering processing on the converted fields to obtain standard access log fields to be detected corresponding to each of the target access logs to be detected; wherein, the standard access log fields to be detected include remote address field, log request field, log status code field, user source field, time field, Internet protocol address field, and access path field.

[0077] Based on the above embodiments, the standard storage access log field determination module 230 can be specifically used to: perform index matching analysis on the time field, Internet protocol address field, access path field, and log status code field in each of the standard access log fields to be detected for anomaly, according to the log index configuration and storage rules; if the index matching analysis result meets the log field storage conditions, then store each of the remote address field, log request field, log status code field, user source field, time field, Internet protocol address field, and access path field in the target log field repository to obtain the stored standard storage access log fields to be detected for anomaly; wherein, different applications correspond to different Internet protocol address fields, and the different field types corresponding to different applications are stored in different locations in the target log field repository; if the index matching analysis result does not meet the log field storage conditions, then a storage error instruction is fed back to the user.

[0078] Based on the above embodiments, the multi-application access log anomaly detection system further includes a log aggregation and summarization module.

[0079] Based on the above embodiments, the standard summary statistical result determination module can be specifically used to: analyze and store each of the standard access log fields to be detected for anomaly detection through the log index configuration and storage rules in the log index and storage module, and obtain the stored standard access log fields to be detected for anomaly detection; then, through the log aggregation and summary module, receive the target log aggregation and summary task in real time, and parse the target log aggregation and summary task to obtain the target log aggregation and summary parsing result; wherein, the target log aggregation and summary parsing result includes the target Internet Protocol address field and the target time field; and summarize the corresponding standard access log fields to be detected for anomaly detection according to the target Internet Protocol address field and the target time field to obtain the standard summary statistical results; wherein, the standard summary statistical results include at least one of the following: the statistical results of the number of accesses per minute corresponding to different applications, the statistical results of the frequency ranking of the access Internet Protocol address field, the statistical results of the status code error rate, and the statistical results of the aggregated distribution of the access path field.

[0080] Based on the above embodiments, the access log anomaly detection result generation and feedback module 240 can be specifically used to: parse the received application access log detection task to obtain the target application to be detected and the target time to be detected; filter the target standard storage access log fields to be detected that correspond to the target application to be detected and the target time to be detected from the standard storage access log fields to be detected; use the abnormal behavior detection module to perform anomaly detection on the target standard storage access log fields to be detected, generate the access log anomaly detection result corresponding to the target application, and provide feedback.

[0081] Based on the above embodiments, an access log anomaly alarm processing module is also included, which can be specifically used to: after the abnormal behavior detection module performs anomaly detection on the target standard storage access log fields to be detected, if there is anomaly in the access log fields, the alarm and linkage module in the multi-application access log anomaly detection system is used to call the target alarm mechanism; obtain the target automatic linkage defense strategy corresponding to the target alarm mechanism, so as to realize access log anomaly alarm processing according to the target alarm mechanism, and combine the target automatic linkage defense strategy to perform automated processing operations on the abnormal access logs.

[0082] The access log anomaly detection device for multiple applications provided in this embodiment of the invention can execute the access log anomaly detection method for multiple applications provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0083] Example 3

[0084] Figure 3A schematic diagram of an electronic device 10, which can be used to implement Embodiment 3 of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as an access log anomaly detection method for multiple applications.

[0088] In some embodiments, the multi-application access log anomaly detection method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multi-application access log anomaly detection method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multi-application access log anomaly detection method by any other suitable means (e.g., by means of firmware).

[0089] The method includes: collecting access logs from corresponding application nodes through the log collectors in the log collection module to obtain target access logs to be detected, and sending the target access logs to be detected to the log preprocessing module through the log collection proxy plugin; wherein, the multi-application access log anomaly detection system includes a log collection module, a log preprocessing module, a log indexing and storage module, and an anomaly behavior detection module; the multi-application access log anomaly detection system is used to detect access logs of multiple applications, each application corresponds to an application node, and an application node log collector is deployed on the application node; through the log preprocessing module, the target access logs to be detected are processed according to a pre-set log preprocessing method to obtain standard access log fields to be detected; through the log index configuration and storage rules in the log indexing and storage module, the standard access log fields to be detected are analyzed and stored to obtain stored standard access log fields to be detected; and according to the received application access log detection task, the anomaly behavior detection module performs anomaly detection on the standard stored access log fields to be detected, generates access log anomaly detection results corresponding to each application, and feeds them back.

[0090] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0094] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0095] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0096] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0098] Example 4

[0099] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions, when executed by a computer processor, are used to execute a method for detecting access log anomalies in multiple applications. The method includes: collecting access logs from corresponding application nodes using log collectors in a log collection module to obtain target access logs to be detected; and sending the target access logs to be detected to a log preprocessing module via a log collection proxy plugin. The system for detecting access log anomalies in multiple applications includes a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module. The system is used to detect access logs from multiple applications. The detection process involves assigning an application node to each application and deploying an application node log collector on each node. A log preprocessing module processes the target access logs to be detected according to a pre-defined log preprocessing method, resulting in standard access log fields to be detected. The log indexing and storage module then analyzes and stores these standard access log fields to be detected, resulting in stored standard access log fields to be detected. Finally, based on the received application access log detection task, an anomaly detection module performs anomaly detection on these standard stored access log fields, generating and feeding back the access log anomaly detection results for each application.

[0100] Of course, the computer-executable instructions provided in the embodiments of the present invention, which include a computer-readable storage medium, are not limited to the method operations described above, but can also perform related operations in the access log anomaly detection for multiple applications provided in any embodiment of the present invention.

[0101] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0102] It is worth noting that in the above embodiments of access log anomaly detection for multiple applications, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting access log anomalies across multiple applications, characterized in that, include: The log collection module collects access logs from each application node log collector to obtain access logs for each target to be detected for anomalies. The log collection proxy plugin then sends the access logs for each target to be detected for anomalies to the log preprocessing module. The multi-application access log anomaly detection system includes a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module. The multi-application access log anomaly detection system is used to detect the access logs of multiple applications. Each application corresponds to an application node, and an application node log collector is deployed on the application node. The log preprocessing module processes the access logs of each target to be detected for anomalies according to the pre-set log preprocessing method to obtain the access log fields of each standard to be detected for anomalies. By configuring the log index and storage rules in the log index and storage module, the standard access log fields to be detected for anomalies are analyzed and stored to obtain the stored standard access log fields to be detected for anomalies. Based on the received application access log detection task, the abnormal behavior detection module performs anomaly detection on the fields of the standard storage access logs to be detected, generates the corresponding access log anomaly detection results for each application, and feeds them back.

2. The method according to claim 1, characterized in that, The log preprocessing module processes the target access logs to be detected for anomalies according to a pre-set log preprocessing method to obtain standard access log fields to be detected for anomalies, including: The field parsing sub-methods corresponding to the log preprocessing method are used to parse the fields of each target access log to be detected for anomalies, and the invalid fields are filtered out to obtain the parsed fields of each target access log. The format structuring sub-method corresponding to the log preprocessing method is used to perform structuring transformation on the parsed fields of each target access log, and to perform abnormal format filtering on the transformed fields to obtain the standard abnormal access log fields corresponding to each target abnormal access log. The standard access log fields to be detected for anomalies include a remote address field, a log request field, a log status code field, a user source field, a time field, an Internet protocol address field, and an access path field.

3. The method according to claim 2, characterized in that, The process involves analyzing and storing the standard access log fields to be detected for anomaly detection using log index configuration and storage rules in the log index and storage module, resulting in the stored standard access log fields to be detected for anomaly detection, including: Based on the log index configuration and storage rules, the target distributed search and analysis engine is used to perform index matching analysis on the time field, Internet protocol address field, access path field and log status code field in each of the standard access log fields to be detected for anomalies. If the index matching analysis results meet the log field storage conditions, then the remote address field, log request field, log status code field, user source field, time field, Internet protocol address field and access path field are stored in the target log field repository to obtain the stored standard storage access log fields to be detected by anomaly. Different applications correspond to different Internet Protocol address fields, and the different field types corresponding to different applications are stored in different locations in the target log field repository. If the index matching analysis results do not meet the log field storage conditions, a storage error instruction will be sent to the user.

4. The method according to claim 3, characterized in that, The multi-application access log anomaly detection system also includes a log aggregation and summarization module; After analyzing and storing the standard access log fields to be detected by anomaly detection through the log index configuration and storage rules in the log index and storage module, and obtaining the stored standard access log fields to be detected by anomaly detection, the method further includes: The target log aggregation and summarization task is received in real time through the log aggregation and summarization module, and the target log aggregation and summarization task is parsed to obtain the target log aggregation and summarization parsing result; wherein, the target log aggregation and summarization parsing result includes the target Internet Protocol address field and the target time field; Based on the target Internet Protocol address field and the target time field, the corresponding standard storage access log fields to be detected for anomaly are summarized to obtain the summary statistical results of each standard. The standard summary statistics include at least one of the following: statistics on the number of accesses per minute for different applications, statistics on the frequency ranking of access to Internet Protocol address fields, statistics on status code error rates, and statistics on the aggregated distribution of access path fields.

5. The method according to claim 4, characterized in that, The process of acquiring and detecting application access logs based on the received application access log detection task involves using an abnormal behavior detection module to perform anomaly detection on each standard stored access log field to be detected, generating and feeding back the access log anomaly detection results for each application, including: The received application access log detection task is parsed to obtain the target application to be detected and the target time to be detected. In the standard storage access log fields to be detected for anomalies, filter the target standard storage access log fields to be detected for anomalies that correspond to the target application to be detected and the target time to be detected. The abnormal behavior detection module is used to perform anomaly detection on the target standard storage access log fields to be detected, generate the corresponding access log anomaly detection results for the target application, and provide feedback.

6. The method according to claim 5, characterized in that, After the abnormal behavior detection module performs anomaly detection on the target standard's access log field to be detected, the following is also included: If there are any anomalies in the access log fields, the target alarm mechanism will be invoked through the alarm and linkage module in the multi-application access log anomaly detection system. Obtain the target automatic linkage defense strategy corresponding to the target alarm mechanism, so as to realize the abnormal alarm processing of access logs according to the target alarm mechanism, and combine the target automatic linkage defense strategy to perform automated processing of abnormal access logs.

7. A device for detecting access log anomalies for multiple applications, characterized in that, include: The target anomaly detection access log sending module is used to collect access logs of the corresponding application nodes through the log collectors of each application node in the log collection module, obtain the target anomaly detection access logs, and send the target anomaly detection access logs to the log preprocessing module through the log collection proxy plugin. The multi-application access log anomaly detection system includes a log collection module, a log preprocessing module, a log indexing and storage module, and an abnormal behavior detection module. The multi-application access log anomaly detection system is used to detect the access logs of multiple applications. Each application corresponds to an application node, and an application node log collector is deployed on the application node. The standard access log field determination module is used to process each target access log to be detected according to a pre-set log preprocessing method through the log preprocessing module to obtain each standard access log field to be detected. The standard storage access log field determination module is used to analyze and store each of the standard access log fields to be detected by anomaly detection through the log index and the log index configuration and storage rules in the storage module, so as to obtain the stored standard storage access log fields to be detected by anomaly detection. The access log anomaly detection result generation and feedback module is used to acquire and, based on the received application access log detection task, perform anomaly detection on the fields of the access logs to be detected in each standard storage through the anomaly behavior detection module, generate access log anomaly detection results corresponding to each application, and provide feedback.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a method for detecting access log anomalies for multiple applications as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a method for detecting access log anomalies for multiple applications as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a method for detecting anomalies in access logs for multiple applications according to any one of claims 1-6.

Images