Log detection method, apparatus, device and computer-readable storage medium

Abstract

The present application belongs to the technical field of cloud computing. Disclosed are a log detection method, an apparatus, a device and a computer-readable storage medium. The method comprises: a first device receiving a log detection request sent by a second device, the log detection request comprising identification information of a first process, and the first process being a process running in the first device; on the basis of the identification information of the first process, acquiring a first log generated by the first process; and acquiring a regular expression of the first log, and, on the basis of the regular expression of the first log, determining a first detection result of the first log, the first detection result indicating whether the content of the first log conforms to a reference format described by the regular expression. The present application can automatically and flexibly acquire, simply by means of the identification information of the first process, the first log requiring detection, and then, on the basis of the regular expression of the first log, perform accurate detection on the automatically acquired first log, thus avoiding errors caused by manual log input.

Description

Log detection method, device, equipment and computer-readable storage medium

[0001] This application claims priority to Chinese patent application No. 202311334488.7, filed on October 13, 2023, entitled “A method and device for detecting anomaly”. This application claims priority to Chinese patent application No. 202311773902.4, filed on December 21, 2023, entitled “Log detection method, device, equipment and computer-readable storage medium”. The entire contents of both applications are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a log detection method, apparatus, device, and computer-readable storage medium. Background Art

[0003] With the development of cloud computing technology, the scale of cloud services based on cloud computing technology is growing, and the application of cloud services is becoming more and more extensive. The governance and operation and maintenance of cloud services are usually based on the analysis of information generated during the operation of various cloud service processes. For example, by detecting whether the process logs generated by the process have anomalies, it is possible to determine whether there are any abnormal processes, implement anomaly alarms, and quickly repair the abnormal processes corresponding to the abnormal process logs, thereby achieving cloud service maintenance.

[0004] Summary of the Invention

[0005] This application provides a log detection method, apparatus, device, and computer-readable storage medium to flexibly and accurately detect logs. The technical solution is as follows:

[0006] In a first aspect, a log detection method is provided, which is applied to a first device providing a cloud service, and the method includes: receiving a log detection request sent by a second device, the log detection request including identification information of a first process, the first process being a process running in the first device; based on the identification information of the first process, obtaining a first log generated by the first process; obtaining a regular expression for the first log, and determining a first detection result for the first log based on the regular expression of the first log, the first detection result indicating whether the content of the first log conforms to a reference format described by the regular expression.

[0007] In the present application, the first device receives a log detection request and, based on the identification information of the first process included in the log detection request, automatically obtains the first log to be detected and the regular expression used to detect the first log, and detects the first log based on the regular expression to obtain a detection result. This eliminates the need for manual input of the log to be detected and the regular expression, making the acquisition and detection of the first log more flexible and avoiding errors caused by manual input of the log and regular expression, resulting in more accurate detection results.

[0008] In one possible implementation, obtaining a first log generated by the first process based on identification information of the first process includes: determining indication information of the first log based on the identification information of the first process, the indication information of the first log being used to indicate a path for obtaining the first log; determining the path for obtaining the first log based on the indication information of the first log, and obtaining the first log based on the path for obtaining the first log. By determining the indication information of the first log based on the identification information of the first process, the first log can be accurately obtained based on the path for obtaining the first log determined by the indication information of the first log, thereby ensuring the reliability of obtaining the first log.

[0009] In one possible implementation, determining the indication information of the first log based on the identification information of the first process includes: obtaining indication information and type information of multiple files generated by at least one process running in the first device, the at least one process including the first process; filtering the indication information of the multiple files based on the type information and the identification information of the first process to obtain the indication information of the first log of the first process. Filtering the indication information of the multiple files based on the identification information and type information of the first process can more accurately obtain the required first log. After filtering to obtain the indication information of the first log, the first log is obtained without having to obtain the multiple files based on the indication information of the multiple files generated by the at least one process and then filtering the files. This reduces the cost of the first device obtaining the first log, reduces resource consumption, and improves the efficiency of obtaining the first log.

[0010] In one possible implementation, obtaining indication information and type information of multiple files generated by at least one process running in a first device includes: based on an extended Berkeley packet filter (eBPF) running on the first device, hooking indication information and type information of multiple files generated by at least one process based on a system call operation, where the system call operation includes at least one of a read operation, a write operation, and an open operation. Since the process runs in the user space of the first device, a system call is made to the data in the kernel space through the system call operation. Therefore, the present application uses eBPF to implement cross-space monitoring of the system call operation of the process, thereby ensuring that the indication information and type information of the multiple files generated by the process can be obtained.

[0011] In one possible implementation, the type information includes the file format, the file's read / write mode information, and the file's write frequency information. This type information is used to determine whether each of the multiple files is a log type. Comprehensively determining the file type based on multiple pieces of information, such as the file format, the file's read / write mode information, and the file's write frequency information, ensures the accuracy of the determination result.

[0012] In one possible implementation, before determining the first detection result of the first log based on the regular expression in the first log, the method further includes: intercepting a portion of the first log to obtain a log sample; sending the log sample to a second device; and receiving a regular expression sent by the second device based on the log sample. After the first device intercepts the log sample, it does not need to send the entire first log to the second device. Instead, it can obtain the regular expression sent by the second device by sending a smaller log sample, thereby reducing the consumption of transmission resources between the first and second devices.

[0013] In one possible implementation, the first detection result includes exception information of the first log, where the exception information indicates that the content of the first log does not conform to a reference format. After determining the first detection result of the first log, the method further includes: determining, based on the exception information, reference resource conditions required for the first device to calculate a first exception report for the first log based on the exception information; calculating the first exception report based on the exception information of the first log, based on resources of the first device satisfying the reference resource conditions; and sending the first exception report to the second device. The first device generates the first exception report based on the exception information and sends the first exception report to the second device, thereby reducing resource consumption of the second device for calculating the exception report.

[0014] In one possible implementation, the first exception report includes at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information. The first exception report carries the exception information, enabling the second device to determine an abnormality in the log; the exception alarm enables the second device to issue an alarm based on the first exception report, prompting the user to correct the abnormal process; and the exception handling instruction enables the second device to determine a correction method for the abnormality and promptly correct the abnormal process.

[0015] In one possible implementation, the method further includes: sending exception information to the second device, where the exception information is used by the second device to determine whether an exception exists in the first log. The exception information enables the second device to determine whether an exception exists in the first process, and to generate an exception report based on the exception information for verifying the first exception report, thereby ensuring the accuracy of the first exception report.

[0016] In a second aspect, a log detection method is provided. The method is applied to a second device providing a cloud service, and includes: obtaining identification information of a first process, where the first process is a process running on the first device; and sending a log detection request including the identification information of the first process to the first device, where the log detection request instructs the first device to detect a first log generated by the first process. The second device sends the log detection request including the identification information of the first process to the first device, enabling the first device to automatically obtain the first log to be detected based on the identification information of the first process, thereby achieving flexible and accurate log detection.

[0017] In one possible implementation, after sending a log detection request including identification information of the first process to the first device, the method further includes: receiving a log sample sent by the first device, where the log sample is generated based on the first log; obtaining a regular expression for detecting the first log based on the log sample; and sending the regular expression to the first device. The second device can obtain the regular expression based on the log sample without receiving the entire first log, thereby reducing resource consumption on the second device.

[0018] In one possible implementation, obtaining a regular expression for detecting a first log based on a log sample includes: determining the log type of the first log based on the log sample; and obtaining a regular expression that matches the log type of the first log. Determining the log type of the first log based on the log sample and then obtaining a regular expression that matches the log type of the first log can ensure the accuracy of the detection result of the first log based on the matching regular expression.

[0019] In one possible implementation, obtaining a regular expression for detecting a first log includes: sending a request for obtaining the regular expression to a third device, the third device running a regular expression management model; and receiving a regular expression determined and sent by the third device based on the regular expression management model and the request. By interacting with the third device running the regular expression management model, the required regular expression can be obtained, thereby ensuring that the first device can detect logs based on the regular expression.

[0020] In one possible implementation, after sending a log detection request including identification information of the first process to the first device, the method further includes: receiving exception information of the first log sent by the first device based on the log detection request; and calculating a second exception report for the first log based on the exception information. The second device calculates the second exception report based on the detected exception information, eliminating the need for the second device to perform log detection, thereby improving the efficiency of the second device in obtaining exception reports.

[0021] In one possible implementation, calculating a second exception report for the first log based on the exception information includes: determining an exception keyword in the exception information, where the exception keyword indicates at least one of the cause of the exception and the condition under which the exception occurred; and analyzing the exception keyword to obtain the second exception report. The exception keyword can be used to determine the cause of the exception or the condition under which the exception occurred. Furthermore, by analyzing the cause of the exception or the condition under which the exception occurred, the abnormal situation can be more accurately determined, resulting in a more accurate second exception report.

[0022] In one possible implementation, the method further includes: receiving a first exception report for the first log sent by the first device based on the log detection request; verifying the first exception report based on the second exception report, and determining a target exception report for presentation to a user based on the verification result, wherein the target exception report indicates an anomaly in the first process and the first log. The second device verifies the first exception report sent by the first device based on the second exception report obtained through autonomous calculation, and determines a final target exception report, thereby further ensuring the accuracy of the target exception report presented to the user.

[0023] In one possible implementation, the first exception report, the second exception report, and the target exception report each include at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information. The exception report carries the exception information and can indicate the abnormality of the log; the exception alarm can cause the second device to issue an alarm to remind the user to correct the abnormal process; and the exception handling instruction can cause the second device to determine a correction method for the abnormality and promptly correct the abnormal process.

[0024] In one possible implementation, obtaining identification information of a first process includes: receiving a process detection request input by a user, the process detection request including the identification information of the first process; and obtaining the identification information of the first process from the process detection request. By receiving the process detection request input by the user, the first process to be detected and the identification information of the first process can be determined. The identification information of the first process can be used to autonomously obtain a first log generated by the first process, eliminating the need for the user to manually input the log to be detected, thereby reducing user usage costs and avoiding errors caused by manual log entry by the user, thereby ensuring the accuracy of log detection.

[0025] In a third aspect, a log detection device is provided. The device is applied to a first device providing a cloud service, and the device includes:

[0026] a receiving module, configured to receive a log detection request sent by the second device, the log detection request including identification information of the first process, where the first process is a process running in the first device;

[0027] an acquisition module, configured to acquire a first log generated by the first process based on identification information of the first process;

[0028] The determination module is configured to obtain a regular expression of the first log and determine a first detection result of the first log based on the regular expression of the first log, where the first detection result indicates whether the content of the first log conforms to a reference format described by the regular expression.

[0029] In one possible implementation, an acquisition module is configured to determine indication information of a first log based on identification information of a first process, where the indication information of the first log is used to indicate an acquisition path for the first log; determine an acquisition path for the first log according to the indication information of the first log, and acquire the first log according to the acquisition path.

[0030] In one possible implementation, an acquisition module is used to obtain indication information and type information of multiple files generated by at least one process running in a first device, where the at least one process includes a first process; and filter the indication information of the multiple files according to the type information and identification information of the first process to obtain indication information of a first log of the first process.

[0031] In one possible implementation, an acquisition module is used to hook, based on the eBPF running on the first device, indication information and type information of multiple files generated by at least one process based on a system call operation, where the system call operation includes at least one of a read operation, a write operation, and an open operation.

[0032] In a possible implementation, the type information includes file format, file read / write mode information, and file write frequency information. The type information is used to determine whether the type of each file among the multiple files is a log type.

[0033] In one possible implementation, the determination module is further configured to intercept a partial fragment of the first log to obtain a log sample; the apparatus further includes a sending module configured to send the log sample to the second device; and the receiving module is further configured to receive a regular expression sent by the second device based on the log sample.

[0034] In one possible implementation, the first detection result includes exception information of the first log, and the exception information of the first log indicates that the content of the first log does not conform to the reference format; the determination module is further used to determine, based on the exception information, the reference resource conditions required for the first device to calculate the first exception report of the first log based on the exception information; based on the resources of the first device satisfying the reference resource conditions, the first exception report is calculated according to the exception information of the first log; the sending module is also used to send the first exception report to the second device.

[0035] In a possible implementation, the first exception report includes at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information.

[0036] In a possible implementation, the sending module is further configured to send exception information to the second device, where the exception information is used by the second device to determine whether an exception exists in the first log.

[0037] In a fourth aspect, a log detection device is provided. The device is applied to a second device providing a cloud service, and the device includes:

[0038] an acquisition module, configured to acquire identification information of a first process, where the first process is a process running in the first device;

[0039] The sending module is configured to send a log detection request including identification information of the first process to the first device, where the log detection request instructs the first device to detect a first log generated by the first process.

[0040] In one possible implementation, the apparatus further includes a receiving module, the receiving module being used to receive a log sample sent by the first device, the log sample being generated based on the first log; the acquiring module being used to acquire a regular expression for detecting the first log based on the log sample; and the sending module being used to send the regular expression to the first device.

[0041] In a possible implementation, the acquisition module is configured to determine the log type of the first log according to the log sample; and acquire a regular expression that matches the log type of the first log.

[0042] In one possible implementation, the acquisition module is used to send an acquisition request for acquiring a regular expression to a third device, where a regular expression management model runs; and receive a regular expression determined and sent by the third device based on the regular expression management model and the acquisition request.

[0043] In a possible implementation, the receiving module is further configured to receive exception information of the first log sent by the first device based on the log detection request; the apparatus further includes a determining module configured to calculate a second exception report of the first log based on the exception information.

[0044] In a possible implementation, the determination module is configured to determine abnormal keywords in the abnormal information, where the abnormal keywords indicate at least one of an abnormal cause and an abnormal occurrence condition of the abnormal information; and analyze the abnormal keywords to obtain a second abnormality report.

[0045] In one possible implementation, the receiving module is further used to receive a first exception report of the first log sent by the first device based on a log detection request; the determining module is further used to verify the first exception report based on the second exception report, and determine a target exception report for presentation to the user based on the verification result, where the target exception report indicates that there are exceptions in the first process and the first log.

[0046] In a possible implementation, the first exception report, the second exception report, and the target exception report all include at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information.

[0047] In a possible implementation, the acquisition module is configured to receive a process detection request input by a user, where the process detection request includes identification information of the first process; and acquire the identification information of the first process from the process detection request.

[0048] In a fifth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the log detection method provided by the aforementioned first aspect, second aspect, and any possible implementation of the first aspect or the second aspect.

[0049] In a sixth aspect, embodiments of the present application provide a computer program product comprising instructions. When the instructions are executed by a computing device cluster, the computing device cluster executes the log detection method provided in the first aspect, the second aspect, and any possible implementation of the first aspect or the second aspect. The computer program product may be a software installation package. When the functions of the computing device cluster described above are required, the computer program product may be downloaded and executed on the computing device cluster.

[0050] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the log detection method provided in the first aspect, the second aspect, and any possible implementation of the first aspect or the second aspect. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0051] It should be understood that the beneficial effects achieved by the technical solutions of the third to seventh aspects of this application and the corresponding possible implementation methods can be referred to the technical effects of the log detection method provided for the first aspect, the second aspect, and any possible implementation method of the first aspect or the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] FIG1 is a schematic diagram of an implementation scenario provided by an embodiment of the present application;

[0053] FIG2 is a schematic diagram of another implementation scenario provided by an embodiment of the present application;

[0054] FIG3 is a flow chart of a log detection method provided in an embodiment of the present application;

[0055] FIG4 is a schematic diagram of an implementation scenario of log detection provided in an embodiment of the present application;

[0056] FIG5 is a flow chart of a log detection method provided in an embodiment of the present application;

[0057] FIG6 is a flow chart of a method for determining a log-type file according to an embodiment of the present application;

[0058] FIG7 is a schematic diagram of the structure of a log detection device provided in an embodiment of the present application;

[0059] FIG8 is a schematic structural diagram of another log detection device provided in an embodiment of the present application;

[0060] FIG9 is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0061] FIG10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0062] FIG11 is a schematic diagram of a connection method for a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0064] With the development of cloud computing technology, the variety and scale of cloud services provided by cloud computing have also increased, and the service ecosystem of cloud services running on cloud computing has also developed. However, this rapid development has brought challenges to the governance of cloud services. For example, log monitoring of cloud services is a key means of achieving cloud service governance and operation and maintenance. Given the large-scale and diverse nature of cloud services, traditional manual log monitoring and problem location techniques are no longer sufficient to cope with the increasingly complex and massive log data generated during cloud service operation. Therefore, log monitoring and analysis are needed to promptly detect abnormal logs and implement maintenance and exception handling for cloud services.

[0065] During the log detection process, the log detection system or log detection system needs to analyze and detect the logs to be detected based on the log path to be detected or the complete log file to be detected and the machine learning algorithm, so as to filter out a small amount of concise and key log information, and automatically provide relevant guidance for cloud service maintenance and exception handling.

[0066] In the related art, users are usually required to manually input relevant information of the log to be detected, such as log path, log sample, content extraction field, regular expression and other information, and then detect and analyze the log to be detected. The operation of the related art is relatively cumbersome, the user's usage cost and price are high, and the user's operation may have errors, affecting the accuracy of log detection. In addition, in the related art, the data of the log to be detected is connected to a unified detection and analysis platform, and the unified detection and analysis platform detects and analyzes the log to be detected, so that the log to be detected occupies the storage resources of the unified detection and analysis platform, which is prone to data accumulation, and the unified detection platform consumes more computing power to detect and analyze a large number of logs to be detected, resulting in reduced performance.

[0067] The present application embodiment provides a kind of detection method of log, and this method can reduce the use cost of the user, flexibly, efficiently and accurately detects log, and can avoid data accumulation.Exemplary, referring to Fig. 1, a schematic diagram of a possible implementation scenario of the method is shown.This implementation scenario includes a first device 11 and a second device 12, and the first device 11 and the second device 12 can be connected by wired or wireless communication.The first device 11 and the second device 12 are both devices that can provide cloud services, for example, in a log detection system or a log detection system, the first device 11 can be a service node (service node), for obtaining the log to be detected and detecting the log, and the second device 12 can be a management surface (management) platform, for interacting with the user, and managing the log detection system or the log detection system.

[0068] Optionally, referring to Figure 2, a schematic diagram of another implementation scenario of the method is shown. The implementation scenario includes not only a first device 11 and a second device 12, but also a third device 13 that can be connected to the second device 12. Among them, the functions of the first device 11 and the second device 12 can refer to the above description of Figure 1 and will not be repeated here. A regular expression management model, such as a large model (LM), can be run on the third device 13. The third device 13 can manage the regular expression for detecting logs through the regular expression management model, and provide the required regular expression to the second device 12, so that the first device 11 and the second device 12 can jointly realize the detection of logs.

[0069] 3 , which shows a flow chart of a log detection method. The method can be applied to the implementation scenario shown in FIG. 1 or FIG. 2 . The method includes but is not limited to the following S301 to S305 .

[0070] S301: The second device obtains identification information of a first process, where the first process is a process running in the first device.

[0071] In the present application, the log is detected by the first device and the second device to determine whether there is an abnormality in the log, which can be used to determine whether there is an abnormality in the process that generates the log or the cloud service to which the process belongs. In an embodiment of the present application, by specifying the process that needs to be detected, the log generated by the process can be automatically obtained, and the obtained log can be detected, and then it can be determined whether there is an abnormality in the process based on the detection result of the log. Therefore, before performing the log detection, the identification information of the process that needs to be detected can be obtained to determine the log that needs to be detected. The process that needs to be detected can be any one or more processes running in the first device, and any one or more processes can belong to the same cloud service or different cloud services. The embodiment of the present application takes the first process as an example to exemplify the log detection method.

[0072] Different processes can be distinguished by different identification information. The identification information of a process can be information such as the name or label of the process. Therefore, the second device can determine the first process that needs to be detected by obtaining the identification information of the first process. The embodiment of the present application does not limit the method for the second device to obtain the identification information of the first process. For example, the second device can obtain the identification information of each process running in the first device, and use one or more processes in each process as the first process in different time periods, and then automatically determine the identification information of the first process from the identification information of each process.

[0073] Alternatively, the second device may also obtain the identification information of the first process through interaction with the user. For example, the second device may receive a process detection request input by the user, the process detection request including the identification information of the first process, and obtain the identification information of the first process from the process detection request. Alternatively, the user may input the process detection request through text input or by selecting a control corresponding to the first process.

[0074] Alternatively, the process detection request may include identification information of a cloud service that the user is interested in, and the second device may determine identification information of a process belonging to the cloud service based on the identification information of the cloud service.

[0075] See Figure 4, which illustrates a log detection implementation scenario. The management plane platform is the second device, and the service node is the first device. A user can input a process detection request to the management plane front-end within the management plane platform. The user specifies the processes or cloud services to which the processes belong that require log anomaly detection and analysis. After receiving the process detection request, the management plane front-end transmits it to the management plane backend, which analyzes the request and obtains the identification information of the first process in the request.

[0076] S302: The second device sends a log detection request including identification information of the first process to the first device, where the log detection request instructs the first device to detect a first log generated by the first process.

[0077] After obtaining the identification information of the first process, the second device determines that it can detect the first log generated by the first process and, based on whether the first log contains an anomaly, determines whether the first process is abnormal, thereby detecting the first process. The second device can then generate a log detection request including the identification information of the first process and send the log detection request to the first device running the first process, instructing the first device to detect the first log.

[0078] For example, referring to the implementation scenario shown in FIG4 , after the management plane backend obtains the identification information of the first process, it generates a log detection request and sends the log detection request, including the process information, to the Log Agent component of the service node, thereby dispatching the log detection task. The Log Agent component, as a daemon process, begins running when the service node is started. This means that the Log Agent component automatically starts running after the service node is started.

[0079] S303: The first device receives a log detection request sent by the second device. The log detection request includes identification information of a first process, where the first process is a process running in the first device.

[0080] After the second device sends a log detection request to the first device, the first device can receive the log detection request, accept the log detection task issued by the second device, and determine that the process to be detected is the first process running in the first device based on the identification information of the first process.

[0081] S304: The first device obtains a first log generated by the first process based on the identification information of the first process.

[0082] The first device may determine that it is necessary to detect the first log generated by the first process, and then obtain the first log based on the identification information of the first process. The embodiment of the present application does not limit the way in which the first device obtains the first log. For example, the indication information of the first log may be determined based on the identification information of the first process, and the acquisition path of the first log may be determined based on the indication information of the first log, and the first log may be obtained based on the acquisition path. The indication information of the first log is used to indicate the acquisition path of the first log, and the indication information of the first log may be the name or storage path of the first log, etc.

[0083] In one possible implementation, the first device can obtain indication information and type information of multiple files generated by at least one process running in the first device, where the at least one process includes the first process; thereafter, the first device can filter the indication information of the multiple files based on the type information and identification information of the first process to obtain indication information of the first log of the first process.

[0084] A plurality of different processes may be run within the first device, and each of the plurality of processes may perform different file operation processes, generating different files, indication information, and type information. Optionally, the first device may, based on the eBPF running on the first device, hook the indication information and type information of a plurality of files generated by at least one process based on a system call (syscall) operation, where the system call operation includes at least one of a read operation, a write operation, and an open operation. The read operation may also be referred to as a system call read operation, which may be represented by syscall:read(), the write operation may also be referred to as a system call write operation, which may be represented by syscall:write(), and the open operation may also be referred to as a system call open operation, which may be represented by syscall:open().

[0085] Referring to Figure 5, a flow chart illustrating a log detection method is shown. Taking process x as an example, in a service node, process x running in user space generates multiple files through operations such as system call start, read, and write operations in the operating system (OS). For example, the eBPF hook module in kernel space can perform real-time hooking and event collection on system call read and write operations within the monitoring scope of process x, obtaining relevant information about each file generated by process x, such as indicator information and file type information. Subsequently, the eBPF map module can share the indicator information and type information of each file with the filter module, allowing the filter module to obtain the indicator information and type information of each file generated by process x. The filter module, also known as the log file automatic identification module or the log file filter module, is located in the log agent component. By continuously hooking and mapping the relevant information of each file through eBPF, the filter module can obtain the relevant information of each file generated by each process, requiring minimal resource usage and performance consumption. Furthermore, this method can obtain information about files generated throughout the life cycle of each process, including information about files that have been archived by the process, that is, information about files that have been sealed and no longer modified by the process.

[0086] Exemplarily, the type information of a file may include the file format, the read / write mode information of the file, and the write frequency information of the file. The type information is used to determine whether the type of each file among the multiple files is a log type.

[0087] For example, the first device can determine whether the type of each file is a log type based on the file determination method of the log type as shown in Figure 6. The determination algorithm can be a judgment algorithm or a matching rule algorithm built into the first device. After starting the determination, it can be determined in turn whether the file format of the file is text format, whether the read and write mode of the process to the file conforms to the read and write mode characteristics of the log type file, whether the write frequency of the file conforms to the write frequency characteristics of the log type file, and whether the form of the file (or the content of the file) conforms to the general rules of the log format. If the file meets the above-mentioned various determination conditions, the file can be determined to be a log type file. If the file does not meet any one or more of the above-mentioned conditions, the determination can be ended and it can be determined that the file is not a log type file. Optionally, after the determination is completed, the determination result can be cached.

[0088] In addition, the first device can also cache the correspondence or mapping relationship between the indication information and type information of each file and the identification information of the process that generates the indication information and type information of each file. For example, in the scenario shown in Figure 4 or Figure 5, the filtering module can continuously and dynamically maintain the file information generated by the file operation process of each process in the service node, that is, the indication information and type information and other file-related information, and generate a mapping table. For example, the filtering module can cache the log (log) 1, log 2 and log 3 and other file (other file) related information generated by process x through read / write (r / w) operations, as well as the mapping relationship between the relevant information of process x and the identification information of process x.

[0089] The first device can obtain the mapping relationship between the indication information and type information of each file and the identification information of the process that generates the indication information and type information of each file in the cached mapping table, saving the time of obtaining the indication information and type information, thereby improving the efficiency of log detection.

[0090] The embodiment of the present application does not limit the order in which the first device filters the indication information of multiple files. That is, the first device can first filter the indication information of multiple files based on the identification information of the first process to determine the indication information of each file generated by the first process, and then filter out the indication information of the log type files generated by the first process based on the type information of each file generated by the first process, so as to obtain the identification information of the first log. For example, referring to Figure 4 or Figure 5, a list storing the indication information of each process can be maintained within the filtering module. After obtaining the indication information of each file generated by process x, the filtering module can identify the indication information of each log type file of process x based on the system call behavior of process x.

[0091] Alternatively, the first device may first filter the type information of each file generated by multiple processes to obtain the indication information of the log type files generated by multiple processes, and then perform a secondary filtering based on the identification information of the first process to obtain the indication information of the log type files generated by the first process, thereby obtaining the indication information of the first log.

[0092] After obtaining the indication information of the first log, the first device can obtain the first log according to the acquisition path indicated by the indication information of the first log. For example, referring to Figure 4 or Figure 5, after filtering out the indication information of the first log, the filtering module sends the indication information of the first log to the detection module. The detection module reads the file of the log type according to the acquisition path indicated by the indication information, that is, reads the first log. The detection module can also be called a local log anomaly detection module, which is located in the log agent component.

[0093] S305: The first device obtains a regular expression of the first log, and determines a first detection result of the first log based on the regular expression of the first log, where the first detection result indicates whether content of the first log conforms to a reference format described by the regular expression.

[0094] Regular expressions, also known as information extraction regular expressions or regular expressions, can describe the reference format (i.e., the correct format) of the log content. Regular expressions can be used to determine whether the log content conforms to the reference format, thereby determining whether the log is correct and whether there are any anomalies. Therefore, before testing the first log, it is necessary to obtain the regular expression used to test the first log.

[0095] In one possible implementation, obtaining the regular expression of the first log may include intercepting a portion of the first log to obtain a log sample, and then sending the log sample to the second device to receive the regular expression sent by the second device based on the log sample.

[0096] Optionally, since the content of the first log is usually in text form, a partial fragment of the text of the first log can be intercepted to obtain a log sample. The partial fragment of the text of the first log that is intercepted can be a text fragment that can reflect the type of the first log. The text fragment can be located at the beginning, middle or end of the first log. The embodiment of the present application does not limit the size and position of the partial fragment of the intercepted text. For example, in the scenario shown in Figure 4 or Figure 5, the log sample can be intercepted by the detection module.

[0097] After obtaining the log sample of the first log, the log sample of the first log can be sent to the second device. For example, the log sample can be reported to the management surface background by the detection module shown in Figure 4 or Figure 5. Afterwards, the second device can receive the log sample sent by the first device, and based on the log sample, obtain a regular expression for detecting the first log. In one possible implementation, the regular expressions used to detect different types of logs are different, so the second device can determine the log type of the first log based on the log sample, and then obtain a regular expression that matches the log type of the first log.

[0098] For example, if it is determined through the log sample of the first log that the first log includes logs of log type A and logs of log type B, the second device needs to obtain regular expression a for detecting logs of log type A and regular expression b for detecting logs of log type B.

[0099] The embodiments of the present application do not limit the method by which the second device obtains the regular expression. If the second device has a cached regular expression that matches the log type of the first log, the second device can obtain the required regular expression from the cache. If the second device does not have a cached regular expression that matches the log type of the first log, the second device can obtain the required regular expression by interacting with other devices that can provide regular expressions.

[0100] For example, the second device may send a request to obtain a regular expression to a third device running a regular expression management model, and receive a regular expression determined and sent by the third device based on the regular expression management model and the request. The regular expression management model running on the third device may be a large model. Optionally, the request may include the log type of the first log, so that the third device can determine a regular expression that matches the log type of the first log.

[0101] For example, referring to Figure 4 , the detection module sends a log sample of the first log to the management backend to query the information extraction regular expression corresponding to the first log, that is, to request to obtain the log exception regular expression (obtain log exception regex). After the management backend determines the log type of the first log based on the log sample, the management backend, acting as the interface of the log detection system to the LM, sends a request to obtain the regular expression that matches the log type of the first log to the third device running the LM. In other words, the management backend queries and requests the information extraction regular expression from the third device.

[0102] After receiving the acquisition request, the third device can determine the regular expression required by the second device through the regular expression management model. For example, it can determine a regular expression that matches the log type of the first log based on the log type of the first log in the acquisition request, and feedback the determined regular expression to the second device, enabling the second device to acquire the regular expression. Optionally, after acquiring the regular expression, the second device can cache the regular expression to make it more efficient the next time it acquires the regular expression, and reduce service calls and resource consumption between the third device and the third device.

[0103] Regardless of the manner in which the second device obtains the regular expression, the second device may send the regular expression for responding to the log sample to the first device after obtaining the regular expression, so that the first device can detect the first log.

[0104] In an embodiment of the present application, the first device only needs to send a smaller log sample of the first log to the second device, without sending the entire first log, to obtain a regular expression for detecting the first log, thereby reducing resource consumption of the first device and the second device.

[0105] After the first device obtains the regular expression, it can perform a test on the first log based on the regular expression to obtain a test result. For example, it can determine whether the content of the first log conforms to the reference format described by the regular expression. If the content of the first log conforms to the reference format of the regular expression, the first log can be considered normal. If the content of the first log does not conform to the reference format of the regular expression, the first log can be considered abnormal.

[0106] In one possible implementation, if the first detection result includes exception information of the first log, the exception information of the first log may indicate that the content of the first log does not conform to the reference format. In this implementation, the first device may determine, based on the exception information, the reference resource conditions required for the first device to calculate the first exception report of the first log based on the exception information, and calculate the first exception report based on the exception information of the first log when the resources of the first device meet the reference resource conditions. The reference resource conditions required for calculating the first exception report may be the size of the required resources, and the size of the required resources is positively correlated with the size of the exception information, that is, the larger the volume of the exception information, the larger the resources required, and the less easy it is to meet the reference resource conditions, while the smaller the volume of the exception information, the smaller the resources required, and the easier it is to meet the reference resource conditions.

[0107] For example, the first device may monitor, in real time, the usage and remaining resource size of resources used to calculate and generate an exception report in the log detection system of the first device through a detection module, and automatically identify whether the first device can locally calculate the first exception report offline. If the remaining resource size of the first device used to calculate the first exception report is sufficient for calculating the first exception report, the first device may autonomously calculate the first exception report based on the exception information.

[0108] After calculating the first exception report, the first device can send the first exception report to the second device. Accordingly, the second device can receive the first exception report of the first log sent by the first device based on the log detection request. Optionally, regardless of whether the resources used by the first device to calculate the first exception report meet the reference resource conditions, or in other words, regardless of whether the first device autonomously calculates and sends the first exception report to the second device, the first device can send the extracted exception information to the second device for the second device to determine that there is an exception in the first log. For example, in the scenario shown in Figure 4 or Figure 5, the detection module can integrate the exception information and upload the exception information (report exception info) or report the exception information to the management surface background.

[0109] The second device receives the exception information of the first log sent by the first device based on the log detection request, and can calculate a second exception report for the first log based on the exception information. The embodiment of the present application does not limit the method by which the device calculates the second exception report. For example, the second device can determine the exception keywords in the exception information, analyze the exception keywords, and obtain the second exception report.

[0110] The exception information received by the second device can be a complete log with an exception, or a fragment or exception keyword of the exception log sorted by the first device. The second device can determine the exception keyword of the first log based on the exception information. Among them, the exception keyword indicates at least one of the abnormal cause and the abnormal condition of the abnormal information. The abnormal cause can be, for example, data retrieval abnormality, calculation result abnormality or data storage abnormality, and the abnormal condition can be, for example, the time of abnormal occurrence, the location of the abnormal code or the length of the abnormal code. The second device can obtain a second exception report by analyzing the abnormal keywords. Optionally, the second exception report can include abnormal keywords, or it can include information obtained by summarizing and classifying the abnormal keywords. In addition, the first device can also generate a first exception report in the same way as the second device generates a second exception report.

[0111] In an embodiment of the present application, the first device can estimate whether the resources of the first device support the first device to autonomously calculate and generate an exception report. When the resources of the first device can support autonomous calculation and generation of an exception report, the first device autonomously calculates and obtains a first exception report. When the resources of the first device are insufficient to support autonomous calculation and generation of an exception report, the first device sends exception information to the second device on the central side, and the second device calculates and obtains a second exception report, so as to achieve further performance control in terms of offline detection of log anomalies, avoid the second device from performing centralized calculation of all exception information reported by the first device, alleviate the computing pressure of the second device, and reduce the resource usage of the second device.

[0112] In one possible implementation, the second device may verify the first exception report based on the second exception report, and determine a target exception report for presentation to the user based on the verification result, where the target exception report indicates that an anomaly exists in the first process and the first log. The second device may compare the first exception report and the second exception report to verify the first exception report. If the first exception report and the second exception report are identical, or if the second exception report and the first exception report include the same exception keywords, the verification result may be that the first exception report and the second exception report match.

[0113] The second device can then determine the target exception report to be presented to the user based on the verification results. For example, if the first exception report and the second exception report are identical, either the first exception report or the second exception report can be determined as the target exception report. If the first exception report and the second exception report match, and the second exception report contains more content than the first exception report, the more detailed second exception report can be determined as the target exception report.

[0114] Optionally, the first exception report, the second exception report, and the target exception report may each include at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information. The exception handling instruction, which may also be referred to as an exception handling suggestion or an exception handling opinion, indicates a method for handling an exception in the first process, so as to facilitate efficient correction and maintenance of the abnormal first process through a more accurate method and resolve problems with the first process. The exception alarm, which may also be referred to as an alarm summary, can implement process operation and maintenance alarms, integrating log anomalies into the log detection system as a sub-capability of the alarm system to indicate an anomaly in the first process. The alarm summary may include a summary generated based on matrix indicator alarms such as central processing unit (CPU) alarms and memory alarms. For example, if the number of abnormal logs per unit time exceeds a quantity threshold, the second device may add an exception alarm to the target exception report, triggering the second device to enable a function for notifying the user of the alarm. The unit time and quantity thresholds for triggering the alarm can be set based on experience or user needs, and are not limited in this embodiment of the present application.

[0115] Exemplarily, the second device presents a target exception report to the user on the front-end interface, realizes a visual presentation of the running status of the process, and provides effective suggestions for exception handling to guide the user to repair the abnormal process in a timely manner.

[0116] The above describes the log detection method provided by the embodiment of the present application. The following will compare and illustrate the beneficial effects of the embodiment of the present application in combination with related technologies.

[0117] Related technology 1 provides a docking system that focuses on the centralized presentation of logs. In related technology 1, after the user activates the log detection service, the user is required to manually input the logs to be detected and the log samples used to detect the logs, log paths, regular expressions and other information into the log detection system, and the log detection system presents the logs to be detected in a unified manner. Related technology 1 does not require users to log in to multiple service node terminals to view the detection results for different logs to be detected, which can improve the efficiency of providing log anomalies to operation and maintenance personnel. By performing complex analysis and calculations on the logs, it provides users with query analysis, visualization, alarm, auditing and other capabilities.

[0118] Related Technology 2 provides a detection system focused on log analysis. This enhances Related Technology 1 by using machine learning algorithms to identify key anomalies in connected logs. Ultimately, only this key anomaly information is presented, and guidance on maintenance and correction is provided based on experience.

[0119] In Related Technology 1 and Related Technology 2, a large number of logs to be detected are connected to the unified analysis platform of the detection system, resulting in data accumulation, which not only occupies a large amount of storage resources of the detection system, but also consumes a large amount of computing power and performance of the detection system due to centralized computing. However, the distributed service nodes of this application have a certain computing power, and the service nodes can detect logs, avoiding the consumption of computing power and performance of the detection system by centralized computing, reducing the memory occupied by the detection process, and improving detection efficiency. In addition, the service nodes in this application can only report the abnormal information after filtering to the management platform, without reporting the full amount of logs, reducing the volume of the reported content, and further reducing the consumption of resources on the management platform.

[0120] In addition, both Related Technology 1 and Related Technology 2 require users to manually connect to logs and manually input a large amount of information related to log detection. The operation is relatively cumbersome and the user has a certain usage cost. The present application does not require users to manually connect to logs, nor does it require users to input a large amount of information related to log detection. That is, the source of local log data at the service node does not rely on user input. The user only needs to specify the process to be detected, and the logs generated by the process can be automatically identified, obtained, and detected. By supporting the front end of the log detection system with automatic identification technology, the user's usage cost is reduced, the log acquisition and detection methods are more flexible, and the errors and inaccuracies that may be caused by user operations are avoided, thereby improving the accuracy of detection.

[0121] To sum up, the log detection method provided in the embodiment of the present application can reduce the user's usage cost and price. Based on the identification information of the process to be detected, the log to be detected and the regular expression used to detect the log can be automatically obtained, and the log can be automatically detected, making the detection more flexible and accurate.

[0122] The above describes the log detection method provided by the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a log detection device. The device is used to execute the log detection method executed by the first device or the second device in Figure 3 above through the various modules shown in Figure 7 or Figure 8. As shown in Figure 7, the log detection device for the first device providing cloud services provided by the embodiment of the present application includes the following modules.

[0123] A receiving module 701 is configured to receive a log detection request sent by a second device, where the log detection request includes identification information of a first process, where the first process is a process running in the first device;

[0124] An acquisition module 702 is configured to acquire a first log generated by the first process based on identification information of the first process;

[0125] The determination module 703 is configured to obtain a regular expression for the first log and determine a first detection result for the first log based on the regular expression for the first log, where the first detection result indicates whether the content of the first log conforms to a reference format described by the regular expression.

[0126] In one possible implementation, the acquisition module 702 is used to determine the indication information of the first log based on the identification information of the first process, where the indication information of the first log is used to indicate the acquisition path of the first log; determine the acquisition path of the first log according to the indication information of the first log, and acquire the first log according to the acquisition path.

[0127] In one possible implementation, the acquisition module 702 is used to obtain indication information and type information of multiple files generated by at least one process running in the first device, where the at least one process includes the first process; the indication information of the multiple files is filtered according to the type information and the identification information of the first process to obtain the indication information of the first log of the first process.

[0128] In one possible implementation, the acquisition module 702 is used to hook the indication information and type information of multiple files generated by at least one process based on the extended Berkeley packet filter eBPF running on the first device, where the system call operation includes at least one of a read operation, a write operation, and an open operation.

[0129] In a possible implementation, the type information includes file format, file read / write mode information, and file write frequency information. The type information is used to determine whether the type of each file among the multiple files is a log type.

[0130] In one possible implementation, the determination module 703 is further used to intercept a partial fragment of the first log to obtain a log sample; the apparatus further includes a sending module, which is used to send the log sample to the second device; and the receiving module 701 is further used to receive a regular expression sent by the second device based on the log sample.

[0131] In one possible implementation, the first detection result includes exception information of the first log, and the exception information of the first log indicates that the content of the first log does not conform to the reference format; the determination module 703 is also used to determine, based on the exception information, the reference resource conditions required for the first device to calculate the first exception report of the first log based on the exception information; based on the resources of the first device satisfying the reference resource conditions, the first exception report is calculated according to the exception information of the first log; the sending module is also used to send the first exception report to the second device.

[0132] In a possible implementation, the first exception report includes at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information.

[0133] In a possible implementation, the sending module is further configured to send exception information to the second device, where the exception information is used by the second device to determine whether an exception exists in the first log.

[0134] As shown in FIG8 , the apparatus for detecting logs of a second device providing cloud services provided in an embodiment of the present application includes the following modules.

[0135] An acquisition module 801 is configured to acquire identification information of a first process, where the first process is a process running in a first device;

[0136] The sending module 802 is configured to send a log detection request including identification information of the first process to the first device, where the log detection request instructs the first device to detect a first log generated by the first process.

[0137] In one possible implementation, the apparatus further includes a receiving module, the receiving module being used to receive a log sample sent by the first device, where the log sample is generated based on the first log; an acquiring module 801 being further used to acquire a regular expression for detecting the first log based on the log sample; and a sending module 802 being further used to send the regular expression to the first device.

[0138] In a possible implementation, the acquisition module 801 is configured to determine the log type of the first log according to the log sample; and acquire a regular expression that matches the log type of the first log.

[0139] In one possible implementation, the acquisition module 801 is used to send an acquisition request for acquiring a regular expression to a third device, where a regular expression management model runs; and receive a regular expression determined and sent by the third device based on the regular expression management model and the acquisition request.

[0140] In a possible implementation, the receiving module is further configured to receive exception information of the first log sent by the first device based on the log detection request; the apparatus further includes a determining module configured to calculate a second exception report of the first log based on the exception information.

[0141] In a possible implementation, the determination module is configured to determine abnormal keywords in the abnormal information, where the abnormal keywords indicate at least one of an abnormal cause and an abnormal occurrence condition of the abnormal information; and analyze the abnormal keywords to obtain a second abnormality report.

[0142] In one possible implementation, the receiving module is further used to receive a first exception report of the first log sent by the first device based on a log detection request; the determining module is further used to verify the first exception report based on the second exception report, and determine a target exception report for presentation to the user based on the verification result, where the target exception report indicates that there are exceptions in the first process and the first log.

[0143] In a possible implementation, the first exception report, the second exception report, and the target exception report all include at least one of exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information.

[0144] In a possible implementation, the acquisition module 801 is configured to receive a process detection request input by a user, where the process detection request includes identification information of the first process; and acquire the identification information of the first process from the process detection request.

[0145] It should be understood that the devices provided in FIG. 7 or FIG. 8 above are merely examples of the division of the functional modules described above when implementing their functions. In actual applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the devices and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0146] In addition, in the above-mentioned log detection device, each module can be implemented by software or hardware. For example, the implementation of acquisition module 801 is described below using acquisition module 801 as an example. Similarly, the implementation of other modules can refer to the implementation of acquisition module 801.

[0147] As an example of a software functional unit, the acquisition module 801 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0148] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0149] As an example of a hardware functional unit, acquisition module 801 may include at least one computing device. Alternatively, acquisition module 801 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0150] The multiple computing devices included in acquisition module 801 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 801 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 801 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0151] It should be noted that in other embodiments, acquisition module 801 can be used to execute any step in the log detection method. That is, the steps that acquisition module 801 and other modules are responsible for implementing can be specified as needed. By having acquisition module 801 and other modules respectively implement different steps in the log detection method, the full functionality of the log detection device can be achieved. In addition, the log detection device and the log detection method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0152] The present application also provides a computing device that can be configured as the first device or the second device in the above-mentioned implementation environment. Referring to Figure 9, Figure 9 is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application. As shown in Figure 9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0153] Bus 902 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG9 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 902 may include a path for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, and communication interface 908).

[0154] The processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0155] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0156] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned modules, thereby implementing the log detection method. That is, the memory 906 stores instructions for executing the log detection method.

[0157] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0158] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be configured as the first device or the second device in the above implementation environment.

[0159] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing the log detection method.

[0160] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the log detection method. In other words, the combination of one or more computing devices 900 can jointly execute the instructions for executing the log detection method.

[0161] It should be noted that the memory 906 in different computing devices 900 in the computing device cluster can store different instructions, each for executing part of the functions of the log detection device. In other words, the instructions stored in the memory 906 in different computing devices 900 can implement the functions of one or more modules in different modules.

[0162] In some embodiments, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), among others. FIG. 11 is a schematic diagram of a connection method for a computing device cluster provided in an embodiment of the present application. As shown in FIG. 11 , two computing devices 900 are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0163] It should be understood that the functions of the computing device 900 shown in FIG. 11 may also be performed by multiple computing devices 900 .

[0164] The present application also provides a computer program product including instructions. This computer program product can be software or a program product including instructions that can be run on a computing device or stored on any available medium. When this computer program product is run on at least one computing device, it causes the at least one computing device to execute the log detection method.

[0165] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a log detection method.

[0166] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0167] In this application, the terms "first," "second," and the like are used to distinguish between identical or similar items having substantially the same function or effect. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms. These terms are simply used to distinguish one element from another.

[0168] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0169] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, "plurality of second devices" means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0170] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0171] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed items. The term "and / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this application generally indicates that the associated objects are in an "or" relationship.

[0172] It should also be understood that the terms “if” and “if” may be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting.” Similarly, the phrases “if it is determined that ” or “if [stated condition or event] is detected” may be interpreted to mean “upon determining ” or “in response to determining ” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event],” depending on the context.

[0173] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0174] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated.

[0175] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0176] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the logs and information involved in this application are all obtained with full authorization.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A log detection method, characterized in that: The method is applied to a first device providing a cloud service, and the method includes: receiving a log detection request sent by a second device, wherein the log detection request includes identification information of a first process, where the first process is a process running in the first device; Based on the identification information of the first process, obtaining a first log generated by the first process; A regular expression of the first log is obtained, and a first detection result of the first log is determined according to the regular expression of the first log, where the first detection result indicates whether content of the first log conforms to a reference format described by the regular expression.

2. The method according to claim 1, characterized in that The acquiring, based on the identification information of the first process, a first log generated by the first process includes: Determine, based on the identification information of the first process, indication information of the first log, where the indication information of the first log is used to indicate an acquisition path of the first log; Determine an acquisition path for the first log according to the indication information of the first log, and acquire the first log according to the acquisition path.

3. The method according to claim 2, characterized in that The determining, based on the identification information of the first process, the indication information of the first log includes: Obtaining indication information and type information of multiple files generated by at least one process running in the first device, the at least one process including the first process; The indication information of the multiple files is filtered according to the type information and the identification information of the first process to obtain the indication information of the first log of the first process.

4. The method according to claim 3, characterized in that The obtaining indication information and type information of multiple files generated by at least one process running in the first device includes: Based on the extended Berkeley packet filter eBPF running on the first device, the indication information and type information of the multiple files generated by the at least one process based on the system call operation are hooked, and the system call operation includes at least one of a read operation, a write operation and an open operation.

5. The method according to claim 4, characterized in that The type information includes file format, file read / write mode information, and file write frequency information, and the type information is used to determine whether the type of each file in the multiple files is a log type.

6. The method according to any one of claims 1 to 5, characterized in that: Before determining the first detection result of the first log according to the regular expression of the first log, the method further includes: intercepting a portion of the first log to obtain a log sample; Sending the log sample to the second device; The regular expression sent by the second device based on the log sample is received.

7. The method according to any one of claims 1 to 6, characterized in that: The first detection result includes abnormal information of the first log, where the abnormal information of the first log indicates that content of the first log does not conform to the reference format; After determining the first detection result of the first log, the method further includes: Determine, according to the exception information, a reference resource condition required for the first device to calculate a first exception report of the first log based on the exception information; Based on the fact that the resources of the first device meet the reference resource condition, and according to the exception information of the first log, calculating the first exception report; Send the first exception report to the second device.

8. The method according to claim 7, characterized in that The first exception report includes at least one of the exception information, an exception alarm generated based on the exception information, and an exception handling instruction for the exception information.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: The exception information is sent to the second device, where the exception information is used by the second device to determine that an exception exists in the first log.

10. A log detection method, characterized in that: The method is applied to a second device that provides a cloud service, and the method includes: Obtaining identification information of a first process, where the first process is a process running in a first device; A log detection request including identification information of the first process is sent to the first device, where the log detection request instructs the first device to detect a first log generated by the first process.

11. The method according to claim 10, characterized in that After sending the log detection request including the identification information of the first process to the first device, the method further includes: receiving a log sample sent by the first device, where the log sample is generated based on the first log; Based on the log sample, obtaining a regular expression for detecting the first log; The regular expression is sent to the first device.

12. The method according to claim 11, characterized in that The obtaining, based on the log sample, a regular expression for detecting the first log includes: Determining the log type of the first log according to the log sample; The regular expression matching the log type of the first log is obtained.

13. The method according to claim 11 or 12, characterized in that: The obtaining of a regular expression for detecting the first log includes: Sending an acquisition request for acquiring the regular expression to a third device, where a regular expression management model runs; Receive the regular expression determined and sent by the third device based on the regular expression management model and the acquisition request.

14. The method according to any one of claims 10 to 13, characterized in that: After sending the log detection request including the identification information of the first process to the first device, the method further includes: receiving abnormal information of the first log sent by the first device based on the log detection request; Based on the exception information, a second exception report of the first log is calculated.

15. The method according to claim 14, characterized in that The calculating, based on the exception information, a second exception report of the first log includes: Determining an abnormal keyword in the abnormal information, wherein the abnormal keyword indicates at least one of an abnormal cause and an abnormal occurrence condition of the abnormal information; The abnormal keywords are analyzed to obtain the second abnormality report.

16. The method according to claim 14 or 15, characterized in that The method further comprises: receiving a first abnormality report of the first log sent by the first device based on the log detection request; The first exception report is verified based on the second exception report, and a target exception report for presentation to a user is determined according to a verification result, wherein the target exception report indicates that anomalies exist in the first process and the first log.

17. The method according to any one of claims 14 to 16, characterized in that: The first exception report, the second exception report and the target exception report all include at least one of the exception information, an exception alarm generated based on the exception information and an exception handling instruction for the exception information.

18. The method according to any one of claims 10 to 17, characterized in that: The obtaining identification information of the first process includes: receiving a process detection request input by a user, wherein the process detection request includes identification information of the first process; The identification information of the first process is obtained from the process detection request.

19. A log detection device, characterized in that: The device is applied to a first device providing a cloud service, and the device includes: A receiving module, configured to receive a log detection request sent by a second device, wherein the log detection request includes identification information of a first process, and the first process is a process running in the first device; an acquisition module, configured to acquire a first log generated by the first process based on the identification information of the first process; The determination module is used to obtain a regular expression of the first log, and determine a first detection result of the first log according to the regular expression of the first log, wherein the first detection result indicates whether the content of the first log conforms to a reference format described by the regular expression.

20. A log detection device, characterized in that: The device is applied to a second device that provides a cloud service, and the device includes: An acquisition module, configured to acquire identification information of a first process, where the first process is a process running in a first device; A sending module is used to send a log detection request including identification information of the first process to the first device, wherein the log detection request instructs the first device to detect a first log generated by the first process.

21. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device includes a processor, and the processor is coupled to a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the log detection method according to any one of claims 1 to 18.

22. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the log detection method according to any one of claims 1 to 18.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the log detection method according to any one of claims 1 to 18.

Images