System log processing method and device, equipment and medium

Abstract

The invention discloses a system log processing method and device, equipment and a medium. The method comprises the steps that system logs of all application systems are collected, and hardware resource index data of all the application systems are detected regularly; the method comprises the following steps: regularly extracting system logs in an original log storage component of each application system, determining business index data of each application system according to the extracted system logs, and determining hardware resource index data associated with the business index data, and storing the business index data of each application system and the hardware resource index data associated with the business index data into an index data storage component of each application system. According to the embodiment of the invention, the business index data used for evaluating the business execution condition of each application system and the hardware resource index data associated with the determined business index data used for evaluating the business execution condition of each application system can be determined regularly based on the system log of each application system.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a system log processing method, apparatus, device, and medium. Background Technology

[0002] Enterprises typically have multiple application systems. Each application system can be a software system used to manage a specific business function within the enterprise. Each application system can run on a dedicated server. System logs are generated during the operation of each application system. Each system log can be a text file describing a business processing operation performed by the application system.

[0003] In related technologies, a common system log processing solution involves periodically statistically analyzing the system logs of various application systems to determine business performance metrics for evaluating their performance. However, the performance of each application system is usually closely related to the resource usage of its infrastructure. Current system log processing solutions rely solely on the system logs to determine these metrics, neglecting to consider the resource usage of the infrastructure that is so closely linked to performance. Consequently, they cannot determine the relevant hardware resource metrics and their associated hardware metrics based solely on the system logs, leading to low detection and response capabilities for problems and potential risks within each application system. Summary of the Invention

[0004] This invention provides a system log processing method, apparatus, device, and medium to address the problem that related system log processing schemes do not focus on the resource usage of infrastructure closely related to the business execution of each application system, and cannot determine business indicator data and hardware resource indicator data associated with the business indicator data based on the system logs of each application system. This results in the low detection and response capabilities of each application system to problems and potential risks in each application system.

[0005] According to one aspect of the present invention, a system log processing method is provided, comprising:

[0006] Collect system logs from each application system, store the collected system logs in the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data in the hardware resource indicator data list of each application system.

[0007] Periodically retrieve system logs from the raw log storage component of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system.

[0008] According to another aspect of the present invention, a system log processing apparatus is provided, comprising:

[0009] The log collection module is used to collect system logs from various application systems, store the collected system logs from various application systems into the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data of each application system into the hardware resource indicator data list of each application system.

[0010] The log processing module is used to periodically retrieve system logs from the raw log storage components of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system.

[0011] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0012] At least one processor;

[0013] and a memory communicatively connected to the at least one processor;

[0014] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the system log processing method according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the system log processing method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the system log processing method described in any embodiment of the present invention.

[0017] The technical solution of this invention collects system logs from various application systems, stores the collected system logs in the original log storage component of each application system, and periodically detects hardware resource indicator data of each application system, storing the detected hardware resource indicator data in the hardware resource indicator data list of each application system. It also periodically retrieves system logs from the original log storage component of each application system, determines business indicator data for each application system based on the retrieved system logs, determines the hardware resource indicator data associated with the business indicator data, and stores the business indicator data and associated hardware resource indicator data of each application system in the indicator data storage component of each application system. This solves the problem that related system log processing schemes do not pay attention to the resource usage of infrastructure closely related to the business execution of each application system, and lack [further details needed]. The method, based on the system logs of each application system, determines business indicator data and associated hardware resource indicator data for evaluating the business performance of each application system. This addresses the issue of low detection and response capabilities of each application system to problems and potential risks. The solution involves periodically determining the business indicator data and associated hardware resource indicator data for evaluating the business performance of each application system based on the system logs. This facilitates analysis and detection by application system maintenance personnel based on the business indicator data and associated hardware resource indicator data, enabling them to locate problems and potential risks in each application system, thereby improving the detection and response capabilities of each application system and enhancing the overall operation and maintenance management level of the application systems.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a system log processing method provided in Embodiment 1 of the present invention.

[0021] Figure 2This is a flowchart of a system log processing method provided in Embodiment 2 of the present invention.

[0022] Figure 3 This is a schematic diagram of a system log processing device provided in Embodiment 3 of the present invention.

[0023] Figure 4 A schematic diagram of the structure of an electronic device for implementing the system log processing method of this embodiment of the invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "target," "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising," "including," and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Example 1

[0027] Figure 1 This is a flowchart illustrating a system log processing method according to Embodiment 1 of the present invention. This embodiment is applicable to processing system logs generated by various application systems within an enterprise. The method can be executed by a system log processing device, which can be implemented in hardware and / or software and can be configured in an electronic device installed within the enterprise. The electronic device can be an electronic device installed within the enterprise for processing system logs generated by various application systems within the enterprise. For example... Figure 1 As shown, the method includes:

[0028] Step 101: Collect system logs from each application system, store the collected system logs from each application system in the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data of each application system in the hardware resource indicator data list of each application system.

[0029] Optionally, a company's business includes, but is not limited to: providing services to other companies or individuals, producing software products, and producing hardware products. Each application system can refer to multiple software systems set up within the company. For each application system, the application system manages a specific business function of the company. The application system runs on a dedicated server. System logs are generated during the operation of the application system. Each system log can be a text generated by the application system describing a business processing operation performed by the application system. The business processing operation can be an operation performed by the application system related to the specific business function of the company managed by the application system. The specific business function of the company managed by the application system can be providing specific services to specified other companies, providing specific services to specified individuals, producing software products with specified functions, or producing hardware products with specified functions.

[0030] Optionally, the system log may include the operation amount, operation time, and operation result. The operation amount in the system log can be the amount recorded by the application system during the execution of the business processing operation described in the system log. The operation time in the system log can be the duration recorded by the application system for executing the business processing operation described in the system log. The operation result in the system log can be text recorded by the application system to indicate whether the business processing operation described in the system log was successfully executed. The operation result is either success or failure. A success result indicates that the application system successfully executed the business processing operation described in the system log. A failure result indicates that the application system did not successfully execute the business processing operation described in the system log.

[0031] Optionally, system logs from each application system are collected, and the collected system logs are stored in the original log storage component of each application system. Hardware resource metrics data of each application system are periodically detected, and the detected hardware resource metrics data are stored in the hardware resource metrics data list of each application system. This includes: collecting system logs from each application system through the system log collection component, storing the collected system logs in the original log storage component of each application system, and periodically detecting various hardware resource metrics data of each application system through the hardware resource monitoring component of each application system, storing the detected hardware resource metrics data in the hardware resource metrics data list of each application system.

[0032] Optionally, the electronic device includes raw log storage components for each application system. Each application system's raw log storage component can be a memory used to store the application system's system logs. For each application system, a system log file is configured. The system log file can be a file used to store system logs. The application system stores each generated system log into the system log file. The application system also includes a system log collection component. This system log collection component can be a software module or a hardware module used to collect the application system's system logs.

[0033] Optionally, for each application system, the system log collection component of the application system can be used to detect whether system logs are stored in the system log file of the application system. After each detection that system logs are stored in the system log file of the application system, the system logs stored in the system log file of the application system are obtained and stored in the original log storage component of the application system. Thus, the system logs of the application system are collected through the system log collection component of the application system and the collected system logs are stored in the original log storage component of the application system.

[0034] Optionally, for each application system, the hardware resource metrics can refer to multiple values ​​that can be used to evaluate the resource usage of the application system's infrastructure. The application system's infrastructure can refer to the server hosting the application system. The resource usage of the application system's infrastructure can refer to the usage of the CPU, memory, and disk within the server hosting the application system. The hardware resource metrics can include CPU utilization, memory usage, and disk I / O rate. CPU utilization can be the ratio of the total number of CPUs currently in use on the server to the total number of CPUs configured on the server. Memory usage can be the amount of memory currently in use on the server. Disk I / O rate can be the number of read / write operations that the disk on the server can process per second.

[0035] Optionally, the electronic device includes a hardware resource indicator data list for each application system. This list can be used to store the hardware resource indicator data for each application system. For each application system, a hardware resource monitoring component is included. This component can be a software or hardware module used to periodically monitor the hardware resource indicator data of the application system.

[0036] Optionally, for each application system, the hardware resource monitoring component of the application system can be used to periodically detect the hardware resource indicator data of the application system and store the detected hardware resource indicator data of the application system into the hardware resource indicator data list of the application system.

[0037] Step 102: Periodically retrieve system logs from the original log storage component of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system.

[0038] Optionally, for each application system, the application system's business metrics data can refer to multiple numerical values ​​that can be used to evaluate the application system's business execution performance. Business execution performance can refer to the application system's performance in executing business processing operations. The application system's business metrics data can include business volume, business processing time, and business success rate. Business volume can refer to the total number of system logs generated within a certain period of time. Business processing time can refer to the average operation time among the various system logs generated within a certain period of time. Business success rate can refer to the ratio of the total number of system logs generated within a certain period of time that contain successful operation results to the total number of all system logs generated within a certain period of time.

[0039] Optionally, for each application system, the hardware resource indicator data associated with the application system's business indicator data can refer to the application system's hardware resource indicator data that describes the resource usage of the application system's infrastructure related to the business execution status described by the application system's business indicator data. Typically, after periodically retrieving system logs from the application system's raw log storage component and determining the application system's business indicator data based on the retrieved system logs, the latest hardware resource indicator data detected by the application system's hardware resource monitoring component is the hardware resource indicator data associated with the currently determined application system's business indicator data.

[0040] Optionally, system logs from the raw log storage component of each application system are periodically retrieved. Based on the retrieved system logs, business indicator data for each application system is determined, and hardware resource indicator data associated with the business indicator data is identified. The business indicator data and associated hardware resource indicator data of each application system are then stored in the indicator data storage component of each application system. This includes performing the following operations for each application system: periodically retrieving system logs from the raw log storage component of the application system; determining the application system's business indicator data based on the retrieved system logs; storing the application system's business indicator data in the application system's business indicator data list; obtaining the latest hardware resource indicator data detected by the application system's hardware resource monitoring component from the application system's hardware resource indicator data list; identifying the obtained hardware resource indicator data as the hardware resource indicator data associated with the application system's business indicator data; and storing the application system's business indicator data and associated hardware resource indicator data accordingly in the application system's indicator data storage component.

[0041] Optionally, system logs can be retrieved from the application system's raw log storage component at preset time intervals. The operation of retrieving all system logs from the application system's raw log storage component can be performed once every preset time interval. The preset time interval can be a pre-set time interval. For example, the preset time interval can be 5 minutes, 10 minutes, 30 minutes, or 1 hour.

[0042] Optionally, the electronic device may contain a list of business indicator data for each application system. This list can be used to store the business indicator data for each application system. After determining the business indicator data for each application system based on retrieved system logs, the determined business indicator data can be stored in the application system's business indicator data list.

[0043] Optionally, business metrics data of the application system can be determined based on the retrieved system logs, including: counting the total number of retrieved system logs to obtain the business volume of the application system; calculating the average operation time in each retrieved system log to obtain the business time consumption of the application system; counting the total number of system logs in each retrieved system logs whose operation result is successful; and calculating the ratio of the total number of system logs in each retrieved system logs whose operation result is successful to the total number of retrieved system logs to obtain the business success rate of the application system.

[0044] Optionally, the latest hardware resource indicator data of the application system stored in the application system's hardware resource indicator data list is the latest hardware resource indicator data of the application system detected by the application system's hardware resource monitoring component. System logs can be retrieved from the application system's raw log storage component at regular intervals. After determining the application system's business indicator data based on the retrieved system logs, the latest hardware resource indicator data of the application system stored in the application system's hardware resource indicator data list can be obtained. This allows us to obtain the latest hardware resource indicator data of the application system detected by the application system's hardware resource monitoring component. This latest hardware resource indicator data is then identified as the hardware resource indicator data associated with the currently determined business indicator data of the application system.

[0045] Optionally, the electronic device includes indicator data storage components for each application system. Each application system's indicator data storage component can be a memory used to store the application system's business indicator data and the hardware resource indicator data associated with that data. After each determination of the application system's business indicator data and the associated hardware resource indicator data, these data can be stored in the application system's indicator data storage component.

[0046] Optionally, it also includes: for each application system, based on the configuration information of the application system and the configuration information of each associated hardware device of the application system, establishing a relationship network diagram corresponding to the application system and each associated hardware device of the application system.

[0047] Optionally, for each application system, the associated hardware devices of the application system can be multiple hardware devices associated with the application system during its operation. These associated hardware devices may include the server hosting the application system, at least one storage device connected to the application system, and at least one network device. Storage devices include, but are not limited to, cloud servers. Network devices include, but are not limited to, routers and switches. The electronic device stores the application system's configuration information. This configuration information can be information describing the application system. It includes the application system's identification information, version information, functional module information, and software environment information. The application system's identification information can be text used to identify the application system. The application system's version information can be information describing the version of the application system. The application system's functional module information can be information describing the functional modules within the application system. The application system's software environment information can be information describing the software environment the application system depends on. The electronic device stores the configuration information of each associated hardware device of the application system. The configuration information of the associated hardware devices can be information describing the associated hardware devices. This configuration information includes the associated hardware device's identification information, model information, CPU model information, memory capacity information, and network interface information. The associated hardware device's identification information can be information used to identify the associated hardware device. The model information of the associated hardware device can be information describing the model of the associated hardware device. The CPU model information of the associated hardware device can be information describing the model of the CPU configured in the associated hardware device. The memory capacity information of the associated hardware device can be information describing the memory capacity configured in the associated hardware device. The network interface information of the associated hardware device can be information describing the network interface configured in the associated hardware device.

[0048] Optionally, the relationship network diagram corresponding to the application system and its associated hardware devices can be a structural diagram representing the relationships between the application system and its associated hardware devices, consisting of application system nodes representing the application system, associated hardware device nodes representing the associated hardware devices, and connecting edges between nodes with relationships. The relationship network diagram corresponding to the application system and its associated hardware devices includes application system nodes representing the application system, associated hardware device nodes representing the associated hardware devices, and connecting edges formed by the relationships between nodes.

[0049] Optionally, for each application system, based on the configuration information of the application system and the configuration information of each associated hardware device of the application system, a relationship network diagram corresponding to the application system and each associated hardware device of the application system is established, including: for each application system, performing the following operations: extracting the identification information of the application system from the configuration information of the application system, and using the identification information of the application system as the application system node to represent the application system; extracting the identification information of each associated hardware device of the application system from the configuration information of each associated hardware device, and using the identification information of each associated hardware device as the associated hardware device node to represent the associated hardware device node; establishing connections between the application system node and each associated hardware device node to obtain the connection edges between the application system node and each associated hardware device node; and aggregating all nodes and the connection edges between nodes to obtain the relationship network diagram corresponding to the application system and the associated hardware devices of the application system.

[0050] Optionally, an application system node is a node used to represent the application system. An associated hardware device node is a node used to represent the associated hardware device. The application system's identification information can be extracted from its configuration information and used as the application system node. For each associated hardware device of the application system, its identification information can be extracted from its configuration information and used as the associated hardware device node. There is an association between the application system and each associated hardware device. The application system node and each associated hardware device node are related nodes. Connections can be established between the application system node and each associated hardware device node to obtain the connection edges between them. After obtaining the connection edges between the application system node, each associated hardware device node, and between the application system node and each associated hardware device node, all nodes and their connection edges are aggregated to obtain a relationship network graph corresponding to the application system and its associated hardware devices.

[0051] Optionally, after establishing the relationship network diagram corresponding to the application system and its various associated hardware devices, the method further includes: updating the relationship network diagram corresponding to the application system and its various associated hardware devices after detecting an update to the configuration information of the application system or the configuration information of the target associated hardware device.

[0052] Optionally, an application system configuration information update can refer to the application system's configuration information being modified by the target user. A target associated hardware device can refer to any one of the application system's associated hardware devices. An update to the configuration information of a target associated hardware device can also refer to the target user modifying the configuration information of that device. The target user can be a technical personnel responsible for managing the application system. The system can detect whether the application system's configuration information and the configuration information of each associated hardware device are updated. After each update is detected, the system updates the relationship network diagram corresponding to each application system and its associated hardware devices. Updating the relationship network diagram can refer to re-establishing the relationship network diagram based on the application system's configuration information and the configuration information of its associated hardware devices.

[0053] Optionally, it also includes: for each application system, based on the service access relationship data of the application system, establishing a relationship network diagram corresponding to each service component and each associated hardware device of the application system.

[0054] Optionally, each application system may have multiple service components. Each service component can be a software module that performs a specified function during the business processing operations of the application system. The specified function implemented by the service component may be to obtain a specified product or to approve whether specified information meets the enterprise's requirements. The electronic device stores service access relationship data of the application system. This service access relationship data may be information compiled by the service component monitoring component set in the application system, describing other service components that have call relationships with each service component and associated hardware devices that have dependencies on each service component. The service component monitoring component set in the application system may be a software module used to obtain the service access relationship data of the application system.

[0055] Optionally, the service access relationship data of the application system can be composed of the service access relationship data of each service component. The service access relationship data of a service component can be information describing other service components that have a calling relationship with the service component and associated hardware devices that have a dependency relationship with the service component. For each service component, the service component's identification information can be information used to identify the service component; other service components that have a calling relationship with the service component can refer to other service components that can be called by the service component or call the service component; and associated hardware devices that have a dependency relationship with the service component can refer to associated hardware devices of the application system that the service component needs to use during operation. When one service component calls another service component, the service component monitoring component records the initiation time of the call, the identification information of the service component initiating the call, the identification information of the called service component, and parameters related to the call process. Based on the recorded information, the service component monitoring component can statistically analyze the service access relationship data of each service component, thereby statistically analyzing the service access relationship data of the application system, and then store the service access relationship data of the application system in the electronic device. The service access relationship data of each service component includes component identification information, calling component identification information, and dependent device identification information. The component identification information is the identification information of the service component. The calling component identifier information is the identifier information of other service components that have a calling relationship with the service component. The dependent device identifier information is the information of the associated hardware devices that have a dependency relationship with the service component.

[0056] Optionally, the relationship network diagram corresponding to the various service components and associated hardware devices of the application system can be a structural diagram representing the call relationships between the various service components of the application system and the dependencies between the various service components and associated hardware devices of the application system. This diagram consists of service component nodes representing the various service components of the application system, associated hardware device nodes representing the various associated hardware devices of the application system, and connection edges formed by call relationships or dependencies between nodes.

[0057] Optionally, for each application system, based on the service access relationship data of the application system, a relationship network diagram corresponding to each service component and each associated hardware device of the application system is established, including: for each application system, performing the following operations: extracting the identification information of each service component of the application system from the service access relationship data of each service component, and using the identification information of each service component as the service component node to represent each service component; extracting the identification information of each associated hardware device of the application system from the configuration information of each associated hardware device, and using the identification information of each associated hardware device as the associated hardware device node to represent each associated hardware device; for each service component, establishing connections between the service component node of the service component and the service component nodes of other service components that have a calling relationship with the service component, and establishing connections between the service component node of the service component and the associated hardware device nodes of the associated hardware devices that have a dependency relationship with the service component, obtaining the connection edges between service component nodes and the connection edges between service component nodes and associated hardware device nodes; and aggregating all nodes and the connection edges between nodes to obtain the relationship network diagram corresponding to each service component and each associated hardware device of the application system.

[0058] Optionally, a service component node is a node used to represent a service component. An associated hardware device node is a node used to represent an associated hardware device. For each service component of the application system, the service component's identification information can be extracted from its service access relationship data, and this identification information is used as the service component node to represent the service component. For each associated hardware device of the application system, the associated hardware device's identification information can be extracted from the application system's configuration information, and this identification information is used as the associated hardware device node to represent the associated hardware device. For each service component of the application system, based on the calling component identification information in the service component's service access relationship data, the service component nodes of other service components with calling relationships with the service component can be determined, and connections can be established between the service component nodes of the service component and the service component nodes of other service components with calling relationships with the service component. Based on the dependent device identification information in the service component's service access relationship data, the associated hardware device nodes of associated hardware devices with dependencies on the service component can be determined, and connections can be established between the service component nodes of the service component and the associated hardware device nodes of associated hardware devices with dependencies on the service component. Thus, the connection edges between service component nodes and the connection edges between service component nodes and associated hardware device nodes are obtained. After obtaining the various service component nodes, associated hardware device nodes, connections between service component nodes, and connections between service component nodes and associated hardware device nodes, all nodes and their connections are aggregated to obtain a relationship network diagram corresponding to the various service components and associated hardware devices of the application system. When an anomaly occurs in the business managed by the application system, the application system's operations and maintenance personnel can quickly locate the problem by consulting the relationship network diagram corresponding to the various service components and associated hardware devices of the application system.

[0059] The technical solution of this invention collects system logs from various application systems, stores the collected system logs in the original log storage component of each application system, and periodically detects hardware resource indicator data of each application system, storing the detected hardware resource indicator data in the hardware resource indicator data list of each application system. It also periodically retrieves system logs from the original log storage component of each application system, determines business indicator data for each application system based on the retrieved system logs, determines the hardware resource indicator data associated with the business indicator data, and stores the business indicator data and associated hardware resource indicator data of each application system in the indicator data storage component of each application system. This solves the problem that related system log processing schemes do not pay attention to the resource usage of infrastructure closely related to the business execution of each application system, and lack [further details needed]. The method, based on the system logs of each application system, determines business indicator data and associated hardware resource indicator data for evaluating the business performance of each application system. This addresses the issue of low detection and response capabilities of each application system to problems and potential risks. The solution involves periodically determining the business indicator data and associated hardware resource indicator data for evaluating the business performance of each application system based on the system logs. This facilitates analysis and detection by application system maintenance personnel based on the business indicator data and associated hardware resource indicator data, enabling them to locate problems and potential risks in each application system, thereby improving the detection and response capabilities of each application system and enhancing the overall operation and maintenance management level of the application systems.

[0060] The technical solution of this invention enables correlation analysis between business performance metrics and hardware resource metrics. This facilitates precise resource allocation during peak business periods based on business performance metrics and their associated hardware resource metrics, preventing resource shortages from impacting business operations and preventing resource idleness during off-peak periods, thus significantly improving resource utilization efficiency. When business anomalies occur, the business performance metrics and their associated hardware resource metrics can quickly pinpoint whether the problem originates from business logic or infrastructure failure, greatly accelerating troubleshooting and reducing recovery time. The precise resource planning and efficient operation and maintenance processes achieved based on business performance metrics and their associated hardware resource metrics can optimize cost structure, reduce enterprise operating costs, and bring comprehensive and substantial improvements to enterprise operation and maintenance management.

[0061] The technical solution of this invention can deeply analyze the close relationship between business usage and infrastructure usage, enabling enterprises to accurately predict resource needs based on actual business load, thereby achieving precise resource allocation and avoiding over- or under-allocation of resources. Resources include, but are not limited to, computing resources and storage resources. This not only reduces waste caused by idle resources but also prevents business malfunctions due to insufficient resources, greatly improving resource utilization. Simultaneously, automated tools can dynamically adjust resource allocation based on actual needs, using business indicator data and associated hardware resource indicator data. When business volume suddenly increases, computing power can be automatically expanded, for example, by increasing the number of CPUs or the memory capacity of the server hosting the application system, ensuring the application system is always in optimal operating condition and effectively coping with changes in resource demand during business peaks and troughs.

[0062] When business operations encounter anomalies, operations and maintenance (O&M) personnel can quickly pinpoint the root cause of the problem based on business performance metrics and associated hardware resource metrics. For example, if the business success rate suddenly drops while abnormally high CPU usage is observed, O&M personnel can quickly determine that the problem likely stems from a server performance bottleneck, rather than blindly searching through business logic and infrastructure failures, significantly shortening the fault location time. Furthermore, by analyzing historical data trends, potential risks can be identified in advance. If the performance metrics of a hardware component are found to be gradually approaching their limits, O&M personnel can take preventative measures in advance, such as replacing hardware or performing performance optimizations, thereby reducing unexpected downtime and ensuring business continuity.

[0063] Precise capacity planning can be achieved based on business performance data and associated hardware resource metrics, effectively reducing unnecessary hardware investment and cloud service costs. Enterprises no longer need to over-allocate resources to cope with potential business peaks, but can dynamically adjust resource scale according to actual business needs, thereby reasonably controlling operating costs.

[0064] Integrating business metrics data with associated hardware resource metrics data significantly simplifies the workflow for operations and maintenance personnel. Previously, information gathering required communication and coordination across multiple application systems and departments was significantly reduced, decreasing manual checks and cross-departmental communication costs. Furthermore, the rapid fault recovery speed achieved based on business metrics data and associated hardware resource metrics data indirectly reduces losses caused by service interruptions. In addition, business metrics data and associated hardware resource metrics data provide enterprises with detailed data support, facilitating analysis of the relationship between business operations and resource usage, and determining whether to introduce new hardware equipment providers or upgrade existing hardware.

[0065] Example 2

[0066] Figure 2 This is a flowchart illustrating a system log processing method according to Embodiment 2 of the present invention. This embodiment of the present invention can be combined with various optional solutions from one or more of the above embodiments. For example... Figure 2 As shown, the method includes:

[0067] Step 201: Collect system logs from each application system, store the collected system logs from each application system in the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data of each application system in the hardware resource indicator data list of each application system.

[0068] Step 202: Periodically retrieve system logs from the original log storage component of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system.

[0069] Step 203: For each application system, based on the configuration information of the application system and the configuration information of each associated hardware device of the application system, establish a relationship network diagram corresponding to the application system and each associated hardware device of the application system.

[0070] Step 204: For each application system, based on the service access relationship data of the application system, establish a relationship network diagram corresponding to each service component and each associated hardware device of the application system.

[0071] The technical solution of this invention can periodically determine business indicator data for evaluating the business execution of each application system based on the system logs of each application system, as well as hardware resource indicator data associated with the determined business indicator data. It can generate a relationship network diagram representing the relationships between application systems and their associated hardware devices, and a relationship network diagram representing the call relationships between service components of the application system and the dependency relationships between service components and associated hardware devices. This facilitates the analysis and detection by application system maintenance personnel based on the business indicator data, associated hardware resource indicator data, and relationship network diagrams. It helps to locate problems and potential risks in each application system, improves the detection and response capabilities of each application system to problems and potential risks, and enhances the operation and maintenance management level of the application system.

[0072] Example 3

[0073] Figure 3 This is a schematic diagram of a system log processing device according to Embodiment 3 of the present invention. The device can be configured in an electronic device. Figure 3 As shown, the device includes a log acquisition module 301 and a log processing module 302.

[0074] The log collection module 301 is used to collect system logs from various application systems, store the collected system logs in the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data in the hardware resource indicator data list of each application system. The log processing module 302 is used to periodically retrieve system logs from the original log storage component of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and the associated hardware resource indicator data of each application system in the indicator data storage component of each application system.

[0075] The technical solution of this invention collects system logs from various application systems, stores the collected system logs in the original log storage component of each application system, and periodically detects hardware resource indicator data of each application system, storing the detected hardware resource indicator data in the hardware resource indicator data list of each application system. It also periodically retrieves system logs from the original log storage component of each application system, determines business indicator data for each application system based on the retrieved system logs, determines the hardware resource indicator data associated with the business indicator data, and stores the business indicator data and associated hardware resource indicator data of each application system in the indicator data storage component of each application system. This solves the problem that related system log processing schemes do not pay attention to the resource usage of infrastructure closely related to the business execution of each application system, and lack [further details needed]. The method, based on the system logs of each application system, determines business indicator data and associated hardware resource indicator data for evaluating the business performance of each application system. This addresses the issue of low detection and response capabilities of each application system to problems and potential risks. The solution involves periodically determining the business indicator data and associated hardware resource indicator data for evaluating the business performance of each application system based on the system logs. This facilitates analysis and detection by application system maintenance personnel based on the business indicator data and associated hardware resource indicator data, enabling them to locate problems and potential risks in each application system, thereby improving the detection and response capabilities of each application system and enhancing the overall operation and maintenance management level of the application systems.

[0076] In an optional embodiment of the present invention, the log collection module 301 is specifically configured to: collect system logs of each application system through the system log collection component of each application system, store the collected system logs of each application system in the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system through the hardware resource monitoring component of each application system, and store the detected hardware resource indicator data of each application system in the hardware resource indicator data list of each application system.

[0077] In an optional embodiment of the present invention, the log processing module 302 is specifically configured to: perform the following operations for each application system: periodically retrieve system logs from the original log storage component of the application system, determine the business indicator data of the application system based on the retrieved system logs, and store the business indicator data of the application system in the business indicator data list of the application system; obtain the latest hardware resource indicator data of the application system detected by the hardware resource monitoring component of the application system from the hardware resource indicator data list of the application system, determine the obtained hardware resource indicator data of the application system as the hardware resource indicator data associated with the business indicator data of the application system; and store the business indicator data of the application system and the hardware resource indicator data associated with the business indicator data of the application system in the indicator data storage component of the application system.

[0078] In an optional embodiment of the present invention, the system log processing device may further include: a first relational network diagram establishment module, configured to establish a relational network diagram corresponding to the application system and its associated hardware devices for each application system, based on the configuration information of the application system and the configuration information of each associated hardware device of the application system.

[0079] In an optional embodiment of the present invention, the first relational network graph establishment module is specifically configured to: perform the following operations for each application system: extract the identification information of the application system from the configuration information of the application system, and use the identification information of the application system as the application system node to represent the application system; extract the identification information of each associated hardware device of the application system from the configuration information of each associated hardware device of the application system, and use the identification information of each associated hardware device as the associated hardware device node to represent the associated hardware device; establish connections between the application system node and each associated hardware device node to obtain the connection edges between the application system node and each associated hardware device node; and aggregate all nodes and the connection edges between nodes to obtain a relational network graph corresponding to the application system and the associated hardware devices of the application system.

[0080] In an optional embodiment of the present invention, the first relationship network graph establishment module is further configured to: update the relationship network graph corresponding to the application system and each associated hardware device of the application system after detecting an update to the configuration information of the application system or the configuration information of the target associated hardware device.

[0081] In an optional embodiment of the present invention, the system log processing device may further include: a second relationship network diagram establishment module, used to establish a relationship network diagram corresponding to each service component and each associated hardware device of the application system for each application system, based on the service access relationship data of the application system.

[0082] The system log processing apparatus provided in this embodiment of the invention can execute the system log processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0083] Example 4

[0084] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement the system log processing method of embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as system log processing methods.

[0088] In some embodiments, the system log processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the system log processing method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the system log processing method by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0091] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0092] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0093] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0094] A computing system can include clients and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and establishing a client-electronic device relationship between them. Electronic devices can be cloud electronic devices, also known as cloud computing electronic devices or cloud servers, which are hosting products within the cloud computing service ecosystem. These address the shortcomings of traditional physical hosting and VPS services, such as high management difficulty and weak business scalability.

[0095] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A system log processing method, characterized in that, include: Collect system logs from each application system, store the collected system logs in the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data in the hardware resource indicator data list of each application system. Periodically retrieve system logs from the raw log storage component of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system.

2. The system log processing method according to claim 1, characterized in that, Collect system logs from various application systems and store them in the original log storage component of each application system. Periodically monitor the hardware resource metrics of each application system and store the detected hardware resource metrics data in the hardware resource metric data list of each application system, including: The system logs of each application system are collected by the system log collection component of each application system and stored in the raw log storage component of each application system. The hardware resource monitoring component of each application system is used to periodically detect the hardware resource indicators of each application system and store the detected hardware resource indicator data in the hardware resource indicator data list of each application system.

3. The system log processing method according to claim 1, characterized in that, Periodically retrieve system logs from the raw log storage components of each application system. Based on the retrieved system logs, determine the business indicator data for each application system, identify the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system, including: Perform the following operations for each application system: Periodically retrieve system logs from the raw log storage component of the application system, determine the business indicator data of the application system based on the retrieved system logs, and store the business indicator data of the application system into the business indicator data list of the application system. The latest hardware resource indicator data of the application system detected by the hardware resource monitoring component is obtained from the hardware resource indicator data list of the application system, and the obtained hardware resource indicator data of the application system is identified as the hardware resource indicator data associated with the business indicator data of the application system. The business indicator data of the application system and the hardware resource indicator data associated with the business indicator data of the application system are stored in the indicator data storage component of the application system.

4. The system log processing method according to claim 1, characterized in that, Also includes: For each application system, a relationship network diagram corresponding to the application system and its associated hardware devices is established based on the application system's configuration information and the configuration information of each associated hardware device.

5. The system log processing method according to claim 4, characterized in that, For each application system, based on the application system's configuration information and the configuration information of its associated hardware devices, a relationship network diagram corresponding to the application system and its associated hardware devices is established, including: Perform the following operations for each application system: Extract the application system's identification information from the application system's configuration information, and use the application system's identification information as the application system node to represent the application system. Extract the identification information of each associated hardware device from the configuration information of each associated hardware device in the application system, and use the identification information of each associated hardware device as the associated hardware device node to represent each associated hardware device. Establish connections between application system nodes and each associated hardware device node to obtain the connection edges between application system nodes and each associated hardware device node; By aggregating all nodes and the connections between them, a relationship network diagram corresponding to the application system and its associated hardware devices is obtained.

6. The system log processing method according to claim 4, characterized in that, After establishing the relationship network diagram corresponding to the application system and its various associated hardware devices, the following is also included: After detecting an update to the configuration information of the application system or the configuration information of the target associated hardware device, update the relationship network diagram corresponding to the application system and each associated hardware device of the application system.

7. The system log processing method according to claim 1, characterized in that, Also includes: For each application system, a relationship network diagram is established based on the service access relationship data of the application system, corresponding to each service component and each associated hardware device of the application system.

8. A system log processing device, characterized in that, include: The log collection module is used to collect system logs from various application systems, store the collected system logs from various application systems into the original log storage component of each application system, and periodically detect the hardware resource indicator data of each application system, storing the detected hardware resource indicator data of each application system into the hardware resource indicator data list of each application system. The log processing module is used to periodically retrieve system logs from the raw log storage components of each application system, determine the business indicator data of each application system based on the retrieved system logs, determine the hardware resource indicator data associated with the business indicator data, and store the business indicator data and associated hardware resource indicator data of each application system into the indicator data storage component of each application system.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the system log processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the system log processing method of any one of claims 1-7.

Images