Log exception processing method, device, equipment and storage medium

Abstract

The present application provides a log exception processing method, device, equipment and storage medium. The present solution is applied to a service system, wherein the log analysis device of the service system obtains logs related to component operation in a data processing device; from multiple keyword sets, a target keyword set matching the log generation source is determined, wherein the target keyword set includes multiple keywords for characterizing that the log generation source has an abnormality; the log is matched with each keyword in the target keyword set; if there is at least one keyword matching the log in the target keyword set, based on the log generation source of the log, an alternative component of the target component or an alternative operating environment of the component operating environment is reconstructed and started in the service system, and the operation of the target component or component operating environment in the data processing device corresponding to the log is terminated. The solution of the present application can timely discover component operation abnormalities and solve component operation abnormalities, and restore the normal operation of the component.

Description

Technical Field

[0001] The present application relates to the field of intelligent technology, and in particular to a log exception processing method, device, equipment and storage medium. Background Art

[0002] Cloud platforms and blockchain and other service systems are generally composed of multiple data processing devices.

[0003] In a service system such as a cloud platform or blockchain, each device generally runs at least one component, and each component has different corresponding functions. If a component in a data processing device runs abnormally, the data processing device will be unable to provide the function corresponding to the component, thereby affecting the functions that can be achieved by the service system, resulting in service abnormalities in the service system. Therefore, how to reduce service abnormalities caused by abnormal operation of components in the service system is a technical problem that technicians in this field need to solve. Summary of the invention

[0004] In view of this, the present application provides a log exception processing method, apparatus, device and storage medium, so as to timely discover component operation abnormalities and resolve component operation abnormalities.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] On the one hand, the present application provides a log exception processing method, which is applied to a log processing device in a service system, wherein the service system includes multiple data processing devices and at least one log processing device, and at least one component is running in the data processing device, including:

[0007] Obtaining a log related to component operation in the data processing device, wherein the log includes log metadata of the log, the log metadata includes a log generation source of the log, and the log generation source represents a target component or a component operation environment to which the log belongs;

[0008] Determine, from a plurality of keyword sets, a target keyword set that matches the log generation source, wherein the target keyword set includes a plurality of keywords for characterizing that the log of the log generation source is abnormal;

[0009] Matching the log with each keyword in the target keyword set;

[0010] If there is at least one keyword matching the log in the target keyword set, based on the log generation source of the log, a substitute component of the target component or a substitute operating environment of the component operating environment is reconstructed and started in the service system, and the operation of the target component or component operating environment in the data processing device corresponding to the log is terminated.

[0011] In a possible implementation, if at least one keyword matching the log exists in the target keyword set, based on a log generation source of the log, reconstructing and starting a substitute component of the target component or a substitute operating environment of the component operating environment in a service system, and ending the operation of the target component or the component operating environment in a data processing device corresponding to the log, includes:

[0012] In the case where there is at least one keyword matching the log in the target keyword set, if the log generation source indicates that the log is a log generated by a target component, starting and running a backup component of the target component in the data processing device, and ending the operation of the target component in the data processing device;

[0013] When there is at least one keyword matching the log in the target keyword set, and if the log generation source indicates that the log is related to the component operating environment, a backup device corresponding to the data processing device is started, and the operation of the data processing device is terminated, and the backup device has the same operating environment as the data processing device.

[0014] In yet another possible implementation, each keyword in the target keyword set has a risk level;

[0015] If at least one keyword matching the log exists in the target keyword set, based on a log generation source of the log, reconstructing and starting a replacement component of the target component or a replacement operating environment of the component operating environment in a service system, and ending the operation of the target component or the component operating environment in a data processing device corresponding to the log, includes:

[0016] If there is at least one keyword matching the log in the target keyword set, and the risk level of the at least one keyword belongs to the set low risk level, the risk number corresponding to the log generation source of the log is increased by one;

[0017] If the number of risks corresponding to the log generation source of the log reaches the set number, a replacement component of the target component or a replacement operating environment of the component operating environment is reconstructed and started in the service system, and the operation of the target component or component operating environment in the data processing device corresponding to the log is terminated.

[0018] In yet another possible implementation, obtaining logs related to component operation in the data processing device includes:

[0019] Obtaining a log queue to be processed, the log queue comprising a plurality of logs from the plurality of data processing devices;

[0020] Before determining a target keyword set matching the log generation source from the plurality of keyword sets, the method further includes:

[0021] According to the sequence of the logs in the log queue, a set number of logs in the log queue are allocated to a set number of log processing threads, so that the set number of log processing threads process the set number of logs in parallel.

[0022] In another aspect, the present application further provides a log exception processing method, which is applied to a service system, wherein the service system includes multiple data processing devices and at least one log processing device, and at least one component is running in the data processing device, and the method includes:

[0023] The data processing device obtains a log related to component operation on the data processing device, and adds log metadata of the log to the log, wherein the log metadata of the log includes a log generation source of the log, and the log generation source represents a target component or component operation environment to which the log belongs;

[0024] The data processing device sends the log with the log metadata added to the log processing device;

[0025] The log processing device determines a target keyword set matching the log generation source from a plurality of keyword sets, wherein the target keyword set includes a plurality of keywords for indicating that the log of the log generation source is abnormal;

[0026] The log processing device matches the log with each keyword in the target keyword set;

[0027] When there is at least one keyword matching the log in the target keyword set, the log processing device reconstructs and starts an alternative component of the target component or an alternative operating environment of the component operating environment in the service system based on the log generation source of the log, and ends the operation of the target component or component operating environment in the data processing device corresponding to the log.

[0028] In another aspect, the present application further provides a log exception processing device, which is applied to a log processing device in a service system, wherein the service system includes a plurality of data processing devices and at least one log processing device, and at least one component is running in the data processing device, including:

[0029] A log obtaining unit, configured to obtain a log related to component operation in the data processing device, wherein the log includes log metadata of the log, the log metadata includes a log generation source of the log, and the log generation source represents a target component or component operation environment to which the log belongs;

[0030] A set determination unit, configured to determine a target keyword set matching the log generation source from a plurality of keyword sets, wherein the target keyword set includes a plurality of keywords for indicating that the log of the log generation source is abnormal;

[0031] A keyword matching unit, used to match the log with each keyword in a target keyword set;

[0032] An exception recovery unit is used to reconstruct and start a substitute component of the target component or a substitute operating environment of the component operating environment in a service system based on a log generation source of the log, if there is at least one keyword matching the log in the target keyword set, and terminate the operation of the target component or component operating environment in the data processing device corresponding to the log.

[0033] On the other hand, the present application also provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the log exception handling method as described in any one of the above items.

[0034] On the other hand, the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the log exception handling method as described in any one of the above items.

[0035] From the above content, it can be seen that in the service system of the present application, the data processing device will report its own logs related to component operation to the log processing device, and the log processing device will perform keyword matching on the log and the keyword set that matches the log generation source of the log. Since the keyword set stores keywords that characterize the log generation source with abnormalities, if the log matches at least one keyword in the keyword set, it means that the log is an abnormal log with abnormalities, and the abnormal log actually characterizes that the component or component operation environment corresponding to the log generation source has abnormalities, so that the component operation abnormalities in each data processing device of the server system can be discovered in time.

[0036] On this basis, when the log matches at least one keyword in the keyword set, the present application will also reconstruct and start a replacement component for the target component corresponding to the log or a replacement operating environment for the component operating environment corresponding to the log in the service system, so as to promptly repair anomalies of certain components or component operating environments in the service system and improve the availability of the service system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0038] Figure 1 A schematic diagram showing the architecture of a service system to which the present application is applicable;

[0039] Figure 2 A flow chart of an embodiment of a log exception processing method provided by the present application is shown;

[0040] Figure 3 A flow chart of another embodiment of a log exception processing method provided by the present application is shown;

[0041] Figure 4 A schematic diagram showing a principle of parallel processing of logs by log processing threads in this application is shown;

[0042] Figure 5 The following is a schematic diagram showing the implementation principle of the log exception handling method of the present application;

[0043] Figure 6 A schematic diagram of a process interaction of the log exception handling method of the present application is shown;

[0044] Figure 7 A schematic diagram showing the composition structure of an embodiment of a log exception processing device of the present application is shown;

[0045] Figure 8 A schematic diagram of the composition architecture of a computer device of the present application is shown. DETAILED DESCRIPTION

[0046] The solution of the present application is applicable to a service system composed of multiple data processing devices, and the service system can be used for data storage or data calculation and other processing.

[0047] For ease of understanding, the following first introduces the architecture of a service system to which the solution of this application is applicable. Figure 1As shown, it shows a schematic diagram of the composition structure of a service system applicable to the present application.

[0048] like Figure 1 As shown, the service system may include: a plurality of data processing devices 101 and at least one log analysis device 102 .

[0049] The data processing device 101 is a device used by the service system to provide services, such as processing business data related to the service system, etc. The log analysis device is a device other than the data processing device that is used to monitor abnormal operation of components in the data processing device, such as the log analysis device can also be commonly known as a peripheral device.

[0050] In the present application, each data processing device can establish a communication connection with the log analysis device through the network.

[0051] Among them, at least one component is installed and running in the data processing device. Among them, a component is a program running in a computer operating system or a group of programs that provide specific services to users. Each component corresponds to a function provided by the data processing device.

[0052] The components in the data processing device run in the component running environment of the data processing device. The component running environment refers to the software environment such as the container and the hardware environment in which the component runs. For example, the component running environment may include a Linux system or a virtual machine.

[0053] It is understandable that the functions that can be implemented by the service system will vary depending on different application requirements.

[0054] For example, the service system may be a cloud service platform, such as a cloud database or a cloud computer platform.

[0055] Correspondingly, the data processing device in the service system may be a cloud server that provides basic cloud computing services such as cloud services, cloud computing, cloud storage, cloud communications, and big data and artificial intelligence platforms.

[0056] Among them, cloud computing services are computing services based on cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0057] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0058] For another example, the service system may be a blockchain system, and on this basis, the data processing device may be a blockchain node in the blockchain.

[0059] Among them, blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0060] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation monitoring. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and the maintenance of the correspondence between the user's real identity and the blockchain address (authority management), etc., and, under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first adapts the interface for parsing and authentication (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration and issuance of contracts, as well as contract triggering and contract execution. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the key or other events are called to trigger the execution and complete the contract logic. It also provides the function of contract upgrade and cancellation. The operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarm, network status monitoring, node equipment health status monitoring, etc.

[0061] The platform product service layer provides the basic capabilities and implementation framework of typical applications. Developers can superimpose business features based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.

[0062] Of course, the service system can also be any system that deploys multiple data processing devices to provide corresponding function implementation through multiple data processing devices.

[0063] In an embodiment of the present application, a log collection program is running in the data processing device. The log collection program can collect logs related to component operation in the data processing device and report the collected logs to the log analysis device.

[0064] In an optional manner, the log analysis device can obtain the log collection rules configured by the user, store the configured log collection rules, and send the log collection rules to the data processing device. The log analysis device can send the latest version of the log collection rules to the data processing device. For example, after the log analysis device detects that the log collection rules have a version update, it sends the latest version of the log collection rules to the data processing device.

[0065] For example, the log analysis device can maintain a connection with the data processing device through a heartbeat mechanism. On this basis, the log analysis device can carry the log collection rules by sending a heartbeat packet to the data processing device. In this case, the log analysis device can send the latest version of the log collection rules through the heartbeat packet. For example, the log analysis device will assign a version number to the configured log collection rules. On this basis, the data processing device can determine whether it already has this version of the log collection rules based on the version number of the log collection rules in the heartbeat packet. If it has not been stored yet, the log collection rules of this version will be saved and the previously saved log collection rules will be deleted.

[0066] The log collection rule includes the type of logs to be collected and the metadata required for the logs, etc. The log metadata included in the configured log collection rule may include the path information of the logs to be collected and the number of the file where the log is located, such as the number of the index node, for example, the inode number in Linux, so that the data processing device can locate the file where the log is located according to the path information and the code and collect it.

[0067] Log types are used to specify which log data related to components or component operating environments need to be collected. For example, log types can be divided into operation log files, output data logs, component test result logs, and operating environment status logs. In actual applications, one or more of these log types can be included in the log collection rules as needed.

[0068] The operation log file records some errors or key information during the operation of the component. Specifically, the component outputs the error information or key information during the operation in the form of text to a specific file. For example, if the component accesses the disk during operation, if the reading is abnormal, an error will be reported and recorded in the specific file. The operation log file can also include operation logs related to the operation environment.

[0069] Of course, the operation log file may also include an operation log file of a component and an operation log file of an operation environment.

[0070] For example, the service system is a cloud database, the components running in the cloud database may include a Structured Query Language (SQL) instance, and the operating environment of the component may be a Linux system. In the scenario, the logs that need to be collected may include SQL error logs and Linux system logs.

[0071] The output data log is the data output during component operation.

[0072] The component test result is the test output obtained by sending an execution command for testing to the component.

[0073] The status log of the operating environment includes a status value representing the status of the operating environment.

[0074] Accordingly, the data processing device may collect logs related to component operation in the data processing device according to the log collection rules, such as collecting logs in the operation log file and component output results.

[0075] In the present application, the log analysis device may also be configured with a keyword set for analyzing log anomalies, wherein the keyword set includes multiple keywords for characterizing the existence of anomalies in the log. From different components or The exceptions in the logs of the operating environment will also be different. Therefore, the present application can configure different keyword sets for different log generation sources. The log generation source represents which component or operating environment the log is generated by.

[0076] Accordingly, the log analysis device may determine a keyword set based on which the anomaly analysis is based in conjunction with the log generation source of the log to be analyzed, and analyze whether anomalies exist in the log based on the keyword set.

[0077] The following is an introduction to the log exception handling method of the present application in conjunction with the flowchart.

[0078] First, the log exception processing method of the present application is introduced from the perspective of the log analysis device. Figure 2 As shown, it shows a flow chart of an embodiment of a log exception processing method of the present application. The method of this embodiment is applied to a log analysis device.

[0079] The method of this embodiment may include:

[0080] S201, obtaining logs related to component operation in a data processing device.

[0081] The log is used to record information related to the operation of the component, and the log can reflect the operating status of the component and possible anomalies. For example, in this application, the log can be a log generated by the component, or a log corresponding to the component operating environment of the component in the data processing device.

[0082] In this application, each log includes log metadata. The log metadata of a log is metadata added by a data processing device to a log collected by the data processing device. Specifically, after obtaining a log related to the operation of a component on the data processing device, the data processing device adds log metadata corresponding to the log to the log.

[0083] Among them, the log metadata can be attribute information such as the collection device that characterizes the log and the source of the log in the computing device. In the present application, the log metadata of the log includes at least the log generation source of the log. The log generation source characterizes the target component or component operating environment to which the log belongs, that is, the log generation source can be used to determine whether the log is a log generated by the component or a log of the component operating environment. Among them, for the sake of ease of distinction, the component that generates the log is called the target component.

[0084] It is understandable that in order to distinguish which data processing device reported the log, the log metadata of the log may also include information about the data processing device that collected the log.

[0085] S202: Determine a target keyword set that matches a log generation source from multiple keyword sets.

[0086] In this application, different keyword sets are configured for different types of components and different component operating environments. Each keyword set includes multiple keywords, which are warning words that indicate that the component or component operating environment corresponding to the keyword set has an abnormality.

[0087] For example, a keyword for the component operating environment may be "device error", which may indicate a disk abnormality.

[0088] Accordingly, since the log generation source of the log can characterize the component or component operating environment to which the log belongs, a keyword set suitable for anomaly detection can be determined in combination with the log generation source. For the sake of distinction, the application refers to the keyword set that matches the log generation source as the target keyword set. The target keyword set includes multiple keywords that characterize the log generation source for the presence of anomalies.

[0089] For example, if the log generation source of the log indicates that the log is generated by the target component, the target keyword set corresponding to the target component can be determined by the category corresponding to the target component. If the log generation source of the log indicates that the log is a log of the component operating environment in the data processing device, the target keyword set corresponding to the component operating environment can be queried.

[0090] S203: Match the log with each keyword in the target keyword set.

[0091] By matching the log with each keyword, the keywords in the log that belong to the target keyword set can be located.

[0092] There are many possible specific implementations of matching the log with each keyword in the target keyword set, and this application does not impose any restrictions on this.

[0093] In an optional manner, in order to improve matching efficiency, the present application may use the AC automaton (Aho-Corasick) algorithm to match the log with each keyword in the target keyword set. Among them, the AC automaton is a string multi-pattern matching algorithm, and keyword matching through this AC automaton algorithm can improve matching efficiency and accuracy.

[0094] S204, if there is at least one keyword matching the log in the target keyword set, based on the log generation source of the log, reconstruct and start an alternative component of the target component or an alternative operating environment of the component operating environment in the service system, and end the operation of the target component or component operating environment in the data processing device corresponding to the log.

[0095] It is understandable that if there is at least one keyword matching the log in the target keyword set, it means that the log hits an abnormal keyword, which means that there is an operation abnormality in the component generating the log or the component operation environment.

[0096] In one possible case, the log generation source characterization log is a log generated by the target component, that is, the log generation source is the target component, then a replacement component of the target component is reconstructed and started in the service system, and the operation of the original target component is terminated.

[0097] In a specific manner, reconstructing and starting a replacement component of a target component in a service system may be: starting and running a backup component of the target component in a data processing device, and correspondingly, the operation of the target component in the data processing device of the log may be collected.

[0098] For example, the data processing device may store backup components of the target component in the form of a component pool, etc. On this basis, if there is an abnormality in the target component, the log analysis device may control the startup of the backup component in the data processing device.

[0099] Alternatively, the log analysis device stores backup component packages of multiple components. Based on this, if it is determined that the target component in the data processing device is abnormal, the backup component package corresponding to the target component can be transmitted to the computer, and the computer can be controlled to start and run the backup component package, so as to start and run the backup component of the target component in the data processing device.

[0100] Of course, the log analysis device can also reconstruct the target component, obtain the data of the backup component of the target component and transmit it to the data processing device.

[0101] In yet another specific embodiment, the service system may reconstruct and start a replacement component of the target component by starting the target component in a data processing device other than the data processing device corresponding to the log to replace the target component running in the data processing device.

[0102] In another possible case, the log generation source characterizing the log is a log generated by the component operating environment, that is, the log generation source is the component operating environment, and then a replacement operating environment of the target component operating environment is reconstructed and started in the service system.

[0103] Specifically, if the log generation source is a component operating environment, a backup device of the data processing device corresponding to the log can be started, and the operation of the data processing device can be terminated. The backup device has the same operating environment as the data processing device.

[0104] By enabling the backup device, the backup device can replace the original data processing device to provide the corresponding services in the service system, thereby restoring the services provided by the original data processing device in the service system in a more timely manner and reducing the user's perception of abnormalities.

[0105] From the above content, it can be seen that in the service system of the present application, the data processing device will report its own logs related to component operation to the log processing device, and the log processing device will perform keyword matching on the log and the keyword set that matches the log generation source of the log. Since the keyword set stores keywords that characterize the log generation source with abnormalities, if the log matches at least one keyword in the keyword set, it means that the log is an abnormal log with abnormalities, and the abnormal log actually characterizes that the component or component operation environment corresponding to the log generation source has abnormalities, so that the component operation abnormalities in each data processing device of the server system can be discovered in time.

[0106] On this basis, when the log matches at least one keyword in the keyword set, the present application will also reconstruct and start a replacement component for the target component corresponding to the log or a replacement operating environment for the component operating environment corresponding to the log in the service system, so as to promptly repair anomalies of certain components or component operating environments in the service system and improve the availability of the service system.

[0107] It can be understood that the above embodiment is described by taking the processing of a log by a log analysis device as an example, however, when there are multiple sets of logs to be processed by the log analysis device at the same time, each log can be processed in sequence according to the method of the above embodiment of the present application, and the specific process is similar.

[0108] In an optional manner, the logs reported by each data processing device in the service system are stored in a log queue, on which the log analysis device can obtain a log queue to be processed. The log queue includes multiple logs from multiple data processing devices. The log queue can be in the order of the logs, for example, the log queue can be a time series.

[0109] Accordingly, the log analysis device may assign a set number of logs in the log queue to a set number of log processing threads according to the sequence of the logs in the log queue, so that the set number of log processing threads process the set number of logs in parallel.

[0110] For each log processing thread, after the log processing thread obtains the assigned log, the process of processing the log can refer to the related operations from S202 to S204 in the previous embodiment, which will not be described in detail.

[0111] It is understandable that for any type of component or component operating environment, in some cases the risk of anomalies in the log is low, or some anomalies in the log may be false alarms. In order to ensure the normal operation of components in the data processing device, the present application can also reconstruct and start the corresponding component or component operating environment after multiple occurrences of anomalies in a component or component operating environment when the anomaly is low risk.

[0112] For easier understanding, see Figure 3 , which shows a flow chart of another embodiment of a log exception processing method of the present application, and the method of this embodiment is applied to a log analysis device. This embodiment may include:

[0113] S301, obtaining a log queue to be processed.

[0114] The log queue includes multiple logs reported by multiple data processing device components in the service system. The log metadata of the log includes the log generation source of the log, and the log generation source represents the target component or component running environment to which the log belongs.

[0115] S302, allocating a set number of logs in the log queue to a set number of log processing threads according to the sequence of the logs in the log queue.

[0116] The logs may be taken out in sequence according to the order of the logs in the log queue and allocated to the idle log processing threads, so that the set number of log processing threads can process the set number of log processing threads in parallel.

[0117] For example, if there are five log processing threads deployed in the log analysis device, five different logs will be processed in parallel by these five log processing threads each time. At the same time, if a thread completes log processing, making the log processing thread idle, a log to be processed will continue to be taken out of the log queue and assigned to the log processing thread.

[0118] It should be noted that this embodiment is described by taking the example of processing logs in parallel through multiple log processing threads. In actual applications, the situation of processing logs one by one is also applicable to this embodiment, and there is no limitation on this.

[0119] S303, for each log processing thread, the log processing thread determines a target keyword set matching the log generation source from multiple keyword sets according to the log generation source corresponding to the log assigned to the log processing thread, and executes step S304.

[0120] In the present application, the target keyword set includes a plurality of keywords used to characterize that the logs of the log generation source are abnormal. At the same time, each keyword in the target keyword set has a risk level.

[0121] For example, the risk level can be divided into a high risk level and a low risk level. Accordingly, according to the actual situation, the risk level of each keyword can be a high risk level or a low risk level.

[0122] Of course, in practical applications, risk levels can be divided into more fine-grained levels, and there is no restriction on this.

[0123] S304: Match the log with each keyword in the target keyword set through the log processing thread.

[0124] For the above steps S303 and S304, reference may be made to the related introduction of the previous embodiment, which will not be described again here.

[0125] S305, if there is at least one keyword matching the log in the target keyword set, detect whether there is a keyword of a set high risk level in the at least one keyword, if yes, execute step S306; if no, execute step S307.

[0126] S306, reconstructing and starting a replacement component of the target component or a replacement operating environment of the component operating environment in the service system, and ending the operation of the target component or the component operating environment in the data processing device corresponding to the log.

[0127] For step S306, reference may be made to the related introduction of the previous embodiment, which will not be described again here.

[0128] S305: Increase the number of risk occurrences corresponding to the log generation source of the log by one.

[0129] S308, if the number of risk occurrences corresponding to the log generation source of the log exceeds the set number, reconstruct and start the alternative component of the target component or the alternative operating environment of the component operating environment in the service system, and end the operation of the target component or the component operating environment in the data processing device corresponding to the log.

[0130] For example, taking the log generation source of the log as the target component as an example, when the at least one keyword matched by the log of the target component does not belong to the set high-risk level keyword, the application determines that the target component has a low risk, and increases the number of risk occurrences of the target component having a low risk by 1. On this basis, the number of risk occurrences can represent the total number of times the target component has currently encountered a low risk.

[0131] It is understandable that for a certain component or component operating environment in a data processing device, the data processing device generally reports the logs of the component and the component operating environment at a set frequency. Therefore, when the number of risk occurrences corresponding to the log generation source (such as the target component or component operating environment) exceeds the set number of times, it means that the frequency of low risks at the log generation source is higher.

[0132] Correspondingly, if the number of risk occurrences corresponding to the log generation source exceeds the set number, it means that there is an abnormality in the target component or component operating environment corresponding to the log generation source, and the abnormality of the target component or component operating environment needs to be repaired.

[0133] For ease of understanding Figure 3 , see Figure 4 , which shows a schematic diagram of the implementation principle of the present application using multi-threaded parallel processing of logs.

[0134] exist Figure 4 Five log processing threads can run in the processor, and these five log processing threads are represented by five thick lines with arrows under the processor.

[0135] exist Figure 4 A small white square in the middle represents a log. Figure 4 Each of the five log processing threads in the processor processes one log.

[0136] exist Figure 4 The squares on the right side of the processor indicate logs that have not yet been assigned to multiple log processing threads for processing. Figure 4 It is shown as a white square to the right of the processor.

[0137] exist Figure 4After the log is processed by the log processing thread, the corresponding abnormal situation of the log is obtained, such as Figure 4 The squares on the left side of the processor indicate the abnormal situations corresponding to the logs. For example, a black square indicates that the log has a high abnormal risk, a square with multiple vertical stripes indicates that the log has a low abnormal risk, and a square with horizontal stripes indicates that the log has no abnormal risk.

[0138] exist Figure 4 In the process, each log processing thread of the processor will match the log with the corresponding keyword set to obtain the keywords matched by the log. Among them, if the keywords matched by the log have high-risk keywords, then the log has a high abnormal risk; similarly, if the keywords matched by the log do not have high risk, but have low-risk keywords, then the log has a low abnormal risk. Correspondingly, if the log does not match the keyword, the log does not have an abnormal risk. In this case, after each log processing thread completes the processing of a log, it will take a log from the logs that have not been processed on the back side of the processor to continue processing.

[0139] For ease of description, the log of a component in a data processing device is taken as an example. The log analysis device can perform different operations based on the matching of the log analyzed by the log processing thread with the keyword, that is, the abnormal risk status of the log. For example, if a log is detected to have a high abnormal risk level, the backup component of the corresponding component in the data processing device can be controlled to start, and the operation of the original component can be shut down. At the same time, if it is analyzed that a component corresponds to a low abnormal risk, the number of logs at a low abnormal risk can be counted, so that when the low abnormal risk of the log corresponding to the component is counted to exceed the set number, the data processing device can be controlled to start the backup component of the component, etc.

[0140] Understandably, Figure 3 The embodiment also belongs to a way of reducing abnormal false alarms. In practical applications, in order to further reduce abnormal false alarms of components or component operating environments, in any of the above embodiments of the present application, before reconstructing and starting a replacement component of the target component or a replacement operating environment of the component operating environment in the service system, the log generation source of the log can also be tested according to the set false alarm test rules.

[0141] Among them, the false alarm test rules may be different for different components or component operating environments. For the same component, if the log of the component matches different keywords, the abnormal conditions of the component will be different, and accordingly, the false alarm test rules may also be different.

[0142] For example, the keywords matched by the component-based log indicate that the component's read and write operations cannot apply for memory. On this basis, instructions for simulating component read and write operations can be sent to the corresponding data processing device to obtain the simulation results. If the simulation results indicate that the content required for component read and write operations can be applied for, it means that the anomaly in the log is a false alarm.

[0143] Correspondingly, if it is determined according to the test results that the abnormality corresponding to the log is a false alarm, the target component or component operating environment corresponding to the target generation source will not be reconstructed and started. If it is determined according to the test results that there is no abnormal false alarm in the log, an operation of reconstructing and starting a replacement component of the target component or a replacement operating environment of the component operating environment in the service system is performed.

[0144] In the present application, the specific method for the data processing device to collect and report logs can be achieved by configuring corresponding log collection rules in the log processing device, and the log processing device sends the corresponding log collection rules to the computer device, thereby controlling the type of logs collected by the computer device.

[0145] For easier understanding, see Figure 5 , Figure 5 A block diagram showing an implementation principle of the log exception processing method of the present application is shown.

[0146] Depend on Figure 5 It can be seen that a configuration module is provided in the service system, and the configuration module is provided in the log processing device. The configuration module can obtain the log collection rules configured by the user, as well as the keyword sets of different components and component operating environments.

[0147] At the same time, the service system is also equipped with a log collection module, wherein a log collection module is provided on each data processing device. The log collection module can obtain the log collection rules configured in the configuration module, and perform log collection based on the log collection rules. The collected logs will be reported to the data processing and alarm module.

[0148] The data processing and alarm module is also provided in the log processing device. The data processing and alarm module can obtain the keyword sets of different components and component operating environments configured from the configuration module. At the same time, the logs reported by the log collection module are detected for anomalies in combination with the keyword sets.

[0149] After the data processing and alarm module detects that the log is abnormal, it will notify the component replacement module to repair the component or component operating environment corresponding to the abnormal log. The component replacement module is set in the log processing device.

[0150] For ease of understanding, the following takes the log collection rules configured in the configuration module as an example to collect logs in the running log file.

[0151] The following describes the log exception processing method of the present application from the perspective of the process interaction between the data processing device and the log processing device. Figure 6 As shown, it shows a process interaction diagram of a log exception processing method of the present application. The method of this embodiment may include:

[0152] S601, the data processing device determines the location point in the running log file where log collection has been completed.

[0153] The operation log file at least includes a log of operation errors during the operation of the component.

[0154] The position point of the completed log collection in the running log file is used to represent the last log that has been collected and processed in the running log file. For example, the position point of the completed log collection is the offset corresponding to the last collected log in the running log file. This position point can be used to represent the starting point of the log to be collected in the running log file.

[0155] In an optional manner, the present application records the location points in the running log file where log collection has been completed through a location point record file. Considering that the file number (such as the index node number) under the same path changes, it means that the file has been rewritten. Therefore, the location point record file in the present application can also record the path information and file number of the log, so that after the file number changes, the log in the file under the path information can be re-read.

[0156] S602: The data processing device obtains uncollected logs from the operation log file according to the location point, and adds log metadata to the logs.

[0157] In an embodiment of the present application, the log metadata may include a log generation source of the log, and the log generation source may be a specific component that generates the log, or a component operating environment that generates the log.

[0158] For example, each log in the operation log file may be associated with information about the component that generated the log, based on which the log generation source of the log may be obtained. Of course, the data processing device may also determine the log generation source corresponding to the log stored by itself in other ways, and this application does not impose any restrictions on this.

[0159] At the same time, in order for subsequent log processing devices to distinguish which data processing device reported the log, the log metadata of the log may also include information about the data processing device that collected the log, such as the device identification of the data processing device.

[0160] It is understandable that in order to subsequently update the location point and enable the log processing device to determine the log collection progress of the data processing device, the log metadata also includes the offset of the log in the running log file.

[0161] S603: The data processing device sends the log with the log metadata to the log processing device.

[0162] It should be noted that in this embodiment, the log type that needs to be collected configured in the configuration module is a log operation file as an example for explanation. In actual applications, the logs that need to be collected may also include the other situations mentioned above.

[0163] For example, the logs to be collected include output data output during the operation of the component. Accordingly, the data processing device obtains the output data output during the operation of the component, determines the output data as a log, and after adding log metadata to the log, can send the log to the log processing device.

[0164] For another example, the logs to be collected include test results obtained when the data processing device uses a test command to run a test on a component. The data processing device obtains the test result, adds an identifier of the component to the test result, and obtains a log.

[0165] S604: The data processing device records the offset of the log in the running log file as the position point in the running log file where log collection has been completed.

[0166] By recording the offset corresponding to the currently reported log, the log that has been collected and reported can be determined. On this basis, no matter why log collection is interrupted, the uncollected log can be determined based on the position of the completed log collection, and log collection can be continued without having to collect logs again from the beginning of the running log file.

[0167] For example, the location point of completed log collection recorded in the location point record file may be updated as the offset of the log.

[0168] It is understandable that, considering that logs may be continuously generated in the data processing device, and the log processing capacity of the log processing device is limited, in order to avoid log accumulation and reduce the processing pressure of the log processing device. If the logs reported by the data processing device and not processed by the log processing device exceed the maximum processing quantity of the log processing device, the logs collected by the data processing device can be discarded, and on this basis, the offset of the collected and discarded logs is also recorded as the position point of the completed log collection.

[0169] S605: The log processing device stores the logs reported by each data processing device in a log queue.

[0170] S606: The log processing device allocates a set number of logs in the log queue to a set number of log processing threads according to the sequence of the logs in the log queue.

[0171] It should be noted that this embodiment is described by taking the example of processing logs in parallel through multiple log processing threads, and the aforementioned situation of processing logs one by one is also applicable to this embodiment.

[0172] S607, for each log processing thread, the log processing thread determines a target keyword set matching the log generation source from multiple keyword sets according to the log generation source corresponding to the assigned log, and executes step S608.

[0173] S608: The log processing device matches the log with each keyword in the target keyword set through the log processing thread.

[0174] S609, when there is at least one keyword matching the log in the target keyword set, such as determining that the log is a log generated by the target component based on the log generation source of the log, according to the information of the data processing device in the log metadata of the log, instruct the data processing device to start and run the backup component of the component, and end the operation of the target component in the data processing device.

[0175] Among them, step S609 is used as an example to illustrate an implementation method of reconstructing and starting the backup component of the target component, and other implementation methods are also applicable to this embodiment.

[0176] S610, when there is at least one keyword matching the log in the target keyword set, such as the log generation source characterization log is a log related to the component operating environment, based on the information of the data processing device in the log metadata of the log, start the backup device corresponding to the data processing device, and end the operation of the data processing device.

[0177] The standby device corresponding to the data processing device has the same operating environment as the data processing device.

[0178] In this embodiment, in order to detect log collection anomalies, the data processing device will also perform self-check on the log collection situation.

[0179] For example, in one possible case, the position point of the completed log collection in the running log file is saved in the position point record file, and the position point of the completed log collection is the offset of the last log of the completed log collection.

[0180] Accordingly, the data processing device can detect the offset difference between the offset of the last log and the size of the running log file. If the offset difference is less than the set value, it is determined that the log collection is abnormal, and an alarm message of the log collection abnormality is sent to the log processing device.

[0181] Among them, if the offset difference between the offset of the last log and the size of the running log file is small, it means that the data processing device starts to collect logs from the starting position of the running log file again, which makes the log collection abnormal. By sending the alarm information of the log collection abnormality to the log processing device, the log processing device can handle the log collection abnormality in time. For example, the log processing device can determine the location point of the collected log based on the log obtained by the log processing device, and instruct the data processing device to collect the log in the running log file according to the location point.

[0182] In addition, when the data processing device detects that the location point of the completed log collection recorded in the location point record file has not been updated for more than a set period of time, it can send a location point update abnormality alarm to the log processing device so that the log processing device can handle the abnormal situation in time.

[0183] Corresponding to the operation on the log processing device side in a log exception processing method of the present application, the present application also provides a log exception processing device.

[0184] like Figure 7 As shown, it shows a flow chart of another embodiment of a log exception processing method of the present application. The method of this embodiment is applied to a log processing device in a service system. The service system includes multiple data processing devices and at least one log processing device. At least one component runs in the data processing device, including:

[0185] A log obtaining unit 701 is used to obtain a log related to component operation in the data processing device, wherein the log includes log metadata of the log, and the log metadata includes a log generation source of the log, and the log generation source represents a target component or component operation environment to which the log belongs;

[0186] A set determining unit 702 is used to determine a target keyword set matching the log generation source from multiple keyword sets, wherein the target keyword set includes multiple keywords used to characterize that the log of the log generation source is abnormal;

[0187] A keyword matching unit 703, used to match the log with each keyword in the target keyword set;

[0188] The abnormal recovery unit 704 is used to reconstruct and start a substitute component of the target component or a substitute operating environment of the component operating environment in the service system based on the log generation source of the log if there is at least one keyword matching the log in the target keyword set, and end the operation of the target component or component operating environment in the data processing device corresponding to the log.

[0189] In a possible implementation, the abnormality recovery unit may include:

[0190] a first exception handling unit, configured to, if at least one keyword matching the log exists in the target keyword set, start and run a backup component of the target component in the data processing device, and terminate the operation of the target component in the data processing device, if the log generation source indicates that the log is a log generated by the target component;

[0191] The second exception handling unit is used to start a backup device corresponding to the data processing device and end the operation of the data processing device when there is at least one keyword matching the log in the target keyword set, and if the log generation source indicates that the log is a log related to the component operating environment, and the backup device has the same operating environment as the data processing device.

[0192] In yet another possible implementation, each keyword in the target keyword set determined by the set determination unit has a risk level;

[0193] The abnormal recovery unit includes:

[0194] A risk accumulation subunit, configured to increase the risk count corresponding to the log generation source of the log by one if there is at least one keyword matching the log in the target keyword set and the risk level of the at least one keyword belongs to a set low risk level;

[0195] The abnormal recovery subunit is used to reconstruct and start a replacement component of the target component or a replacement operating environment of the component operating environment in the service system if the number of risks corresponding to the log generation source of the log reaches a set number, and end the operation of the target component or component operating environment in the data processing device corresponding to the log.

[0196] In yet another possible implementation, the log obtaining unit includes:

[0197] A queue obtaining unit, used for obtaining a log queue to be processed, the log queue including a plurality of logs from the plurality of data processing devices;

[0198] The device also includes:

[0199] The thread allocation unit is used to allocate a set number of logs in the log queue to a set number of log processing threads according to the sequence of the logs in the log queue before determining a target keyword set that matches the log generation source from multiple keyword sets, so that the set number of log processing threads can process the set number of logs in parallel.

[0200] On the other hand, the present application also provides a computer device, which may be the log processing device or the data processing device mentioned above.

[0201] like Figure 8 , which shows a schematic diagram of the composition architecture of a computer device provided by the present application. Figure 8 In the embodiment, the computer device 800 may include: a processor 801 and a memory 802.

[0202] Optionally, the computer device may further include: a communication interface 803 , an input unit 804 , a display 805 and a communication bus 806 .

[0203] The processor 801 , the memory 802 , the communication interface 803 , the input unit 804 and the display 805 all communicate with each other via the communication bus 806 .

[0204] In the embodiment of the present application, the processor 801 may be a central processing unit, a specific application integrated circuit, etc.

[0205] The memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the log exception handling method mentioned in the above embodiment.

[0206] In a possible implementation, the memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, the programs mentioned above, etc.; the data storage area may store data created during the use of the computer device.

[0207] The communication interface 803 may be an interface of a communication module.

[0208] The present application may further include an input unit 804, which may include a touch sensing unit, a keyboard, and the like.

[0209] The display 805 includes a display panel, such as a touch display panel.

[0210] certainly, Figure 8 The computer device structure shown does not constitute a limitation on the computer device in the embodiment of the present application. In actual applications, the computer device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0211] On the other hand, the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the log exception handling method of any one of the above embodiments.

[0212] The present application also proposes a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations of the above-mentioned log exception processing method or log exception processing device. The specific implementation process can refer to the description of the above-mentioned corresponding embodiment, and will not be repeated.

[0213] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. At the same time, the features recorded in each embodiment in this specification can be replaced or combined with each other, so that professionals in the field can implement or use this application. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0214] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0215] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0216] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A log exception processing method, characterized in that: A log processing device applied to a service system, wherein the service system includes a plurality of data processing devices and at least one log processing device, wherein at least one component runs in the data processing device, and the method includes: Obtaining a log related to component operation in the data processing device, wherein the log includes log metadata of the log, the log metadata includes a log generation source of the log, and the log generation source represents a target component or component operation environment to which the log belongs; the log metadata also includes an offset of the log in an operation log file; a position point in the operation log file where log collection has been completed is saved in a position point record file, and the position point where log collection has been completed is an offset of the last log for which log collection has been completed; The data processing device detects the offset difference between the offset of the last log and the size of the running log file, and if the offset difference is less than a set value, it is determined that the log collection is abnormal, and obtains the alarm information of the log collection abnormality sent by the data processing device; When the data processing device detects that the location point for which log collection has been completed and recorded in the location point record file has not been updated for more than a set period of time, obtaining a location point update abnormality alarm sent by the data processing device; Determine, from a plurality of keyword sets, a target keyword set that matches the log generation source, wherein the target keyword set includes a plurality of keywords for characterizing that the log of the log generation source is abnormal; Matching the log with each keyword in the target keyword set; If there is at least one keyword matching the log in the target keyword set, based on the log generation source of the log, a substitute component of the target component or a substitute operating environment of the component operating environment is reconstructed and started in the service system, and the operation of the target component or component operating environment in the data processing device corresponding to the log is terminated.

2. The method according to claim 1, characterized in that: If at least one keyword matching the log exists in the target keyword set, based on a log generation source of the log, reconstructing and starting a replacement component of the target component or a replacement operating environment of the component operating environment in a service system, and ending the operation of the target component or the component operating environment in a data processing device corresponding to the log, includes: In the case where there is at least one keyword matching the log in the target keyword set, if the log generation source indicates that the log is a log generated by a target component, starting and running a backup component of the target component in the data processing device, and ending the operation of the target component in the data processing device; When there is at least one keyword matching the log in the target keyword set, and if the log generation source indicates that the log is related to the component operating environment, a backup device corresponding to the data processing device is started, and the operation of the data processing device is terminated, and the backup device has the same operating environment as the data processing device.

3. The method according to claim 1, characterized in that: Each keyword in the target keyword set has a risk level; If at least one keyword matching the log exists in the target keyword set, based on a log generation source of the log, reconstructing and starting a replacement component of the target component or a replacement operating environment of the component operating environment in a service system, and ending the operation of the target component or the component operating environment in a data processing device corresponding to the log, includes: If there is at least one keyword matching the log in the target keyword set, and the risk level of the at least one keyword belongs to the set low risk level, the risk number corresponding to the log generation source of the log is increased by one; If the number of risks corresponding to the log generation source of the log reaches the set number, a replacement component of the target component or a replacement operating environment of the component operating environment is reconstructed and started in the service system, and the operation of the target component or component operating environment in the data processing device corresponding to the log is terminated.

4. The method according to claim 1, characterized in that The obtaining of logs related to component operation in the data processing device includes: Obtaining a log queue to be processed, the log queue comprising a plurality of logs from the plurality of data processing devices; Before determining a target keyword set matching the log generation source from the plurality of keyword sets, the method further includes: According to the sequence of the logs in the log queue, a set number of logs in the log queue are allocated to a set number of log processing threads, so that the set number of log processing threads process the set number of logs in parallel.

5. A log exception processing method, characterized in that: Applied to a service system, the service system includes a plurality of data processing devices and at least one log processing device, at least one component is running in the data processing device, and the method includes: The data processing device obtains a log related to the operation of a component on the data processing device, and adds log metadata of the log to the log, wherein the log metadata of the log includes a log generation source of the log, and the log generation source represents a target component or component operation environment to which the log belongs; the log metadata also includes an offset of the log in the operation log file; the position point of the completed log collection in the operation log file is saved in a position point record file, and the position point of the completed log collection is the offset of the last log for which the log collection has been completed; The data processing device detects an offset difference between an offset of the last log and a size of the running log file, and if the offset difference is less than a set value, determines that log collection is abnormal, and sends an alarm message of log collection abnormality to the log processing device; When the data processing device detects that the location point of the completed log collection recorded in the location point record file has not been updated for more than a set period of time, the data processing device sends a location point update abnormality alarm to the log processing device; The data processing device sends the log with the log metadata added to the log processing device; The log processing device determines a target keyword set matching the log generation source from a plurality of keyword sets, wherein the target keyword set includes a plurality of keywords for indicating that the log of the log generation source is abnormal; The log processing device matches the log with each keyword in the target keyword set; When there is at least one keyword matching the log in the target keyword set, the log processing device reconstructs and starts an alternative component of the target component or an alternative operating environment of the component operating environment in the service system based on the log generation source of the log, and ends the operation of the target component or component operating environment in the data processing device corresponding to the log.

6. The method according to claim 5, characterized in that The data processing device obtains a log related to the operation of a component on the data processing device, and adds log metadata of the log to the log, including: The data processing device determines a location point in an operation log file where log collection has been completed, wherein the operation log file at least includes a log of operation errors during the operation of the component; According to the location point, obtaining the uncollected log from the operation log file, and adding the log metadata to the log; After the data processing device sends the log with the log metadata added to the log processing device, the method further includes: The data processing device records the offset of the log in the running log file as the position point in the running log file where log collection has been completed.

7. A log exception processing device, characterized in that: A log processing device applied to a service system, wherein the service system includes a plurality of data processing devices and at least one log processing device, wherein at least one component runs in the data processing device, and the apparatus includes: A log acquisition unit, used to obtain logs related to component operation in the data processing device, wherein the logs include log metadata of the logs, the log metadata include a log generation source of the logs, and the log generation source represents a target component or component operation environment to which the logs belong; the log metadata also includes an offset of the log in an operation log file; a position point in the operation log file where log collection has been completed is saved in a position point record file, and the position point where log collection has been completed is the offset of the last log for which log collection has been completed; A set determination unit, configured to determine a target keyword set matching the log generation source from a plurality of keyword sets, wherein the target keyword set includes a plurality of keywords for indicating that the log of the log generation source is abnormal; A keyword matching unit, used to match the log with each keyword in a target keyword set; an abnormality recovery unit, configured to reconstruct and start a replacement component of the target component or a replacement operating environment of the component operating environment in a service system based on a log generation source of the log if at least one keyword matching the log exists in the target keyword set, and terminate the operation of the target component or the component operating environment in a data processing device corresponding to the log; The log processing device is further used to detect the offset difference between the offset of the last log and the size of the running log file in the data processing device, and if the offset difference is less than a set value, it is determined that the log collection is abnormal, and obtain the alarm information of the log collection abnormality sent by the data processing device; The log processing device is also used to obtain a location point update abnormality alarm sent by the data processing device when the data processing device detects that the location point for which log collection has been completed and recorded in the location point record file has not been updated for more than a set period of time.

8. The device according to claim 7, characterized in that The abnormality recovery unit comprises: a first exception handling unit, configured to, if at least one keyword matching the log exists in the target keyword set, start and run a backup component of the target component in the data processing device, and terminate the operation of the target component in the data processing device, if the log generation source indicates that the log is a log generated by the target component; The second exception handling unit is used to start a backup device corresponding to the data processing device and end the operation of the data processing device when there is at least one keyword matching the log in the target keyword set, and if the log generation source indicates that the log is a log related to the component operating environment, and the backup device has the same operating environment as the data processing device.

9. The device according to claim 7, characterized in that Each keyword in the target keyword set determined by the set determination unit has a risk level; The abnormality recovery unit includes: A risk accumulation subunit, configured to increase the risk count corresponding to the log generation source of the log by one if there is at least one keyword matching the log in the target keyword set and the risk level of the at least one keyword belongs to a set low risk level; The abnormal recovery subunit is used to reconstruct and start a replacement component of the target component or a replacement operating environment of the component operating environment in the service system if the number of risks corresponding to the log generation source of the log reaches a set number, and end the operation of the target component or component operating environment in the data processing device corresponding to the log.

10. The device according to claim 7, characterized in that The log obtaining unit includes: A queue obtaining unit, used for obtaining a log queue to be processed, the log queue including a plurality of logs from the plurality of data processing devices; The device also includes: The thread allocation unit is used to allocate a set number of logs in the log queue to a set number of log processing threads according to the sequence of the logs in the log queue before determining a target keyword set that matches the log generation source from multiple keyword sets, so that the set number of log processing threads can process the set number of logs in parallel.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the log exception handling method as described in any one of claims 1-4 or 5-6.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the log exception handling method as described in any one of claims 1-4 or 5-6.

13. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor for a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the log exception handling method as described in any one of claims 1-4 or 5-6.

Images