Log analysis method, device, equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a log analysis method and device, equipment and a storage medium, wherein the method comprises: in response to an analysis request for a to-be-analyzed work order, obtaining a tracking identifier of a call link generated for the to-be-analyzed work order; wherein the call link is used to represent a path of at least two layers of services called for the to-be-analyzed work order; based on the tracking identifier, obtaining real-time log data and offline log data associated with the call link from each layer of service respectively; and analyzing the real-time log data and the offline log data respectively to obtain an analysis result of the to-be-analyzed work order. Embodiments of the present disclosure can simply and accurately obtain the analysis result of the to-be-analyzed work order, simplify the log analysis process, reduce the workload of the user, and improve the accuracy and efficiency of log analysis.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of computer technology, and in particular to a log analysis method, apparatus, device, and storage medium. Background Technology

[0002] Currently, work order analysis systems are mostly used for process control, but in practice, a large number of operators still need to be involved in the analysis. However, different operators have different standards, which can lead to problems such as incorrect work order transfers, inaccurate identification of key root causes, and duplicate work orders. This results in slow processing, low efficiency, a large amount of ineffective work, and low customer satisfaction. Summary of the Invention

[0003] In view of this, the present disclosure provides at least one log analysis method, apparatus, device, and storage medium.

[0004] The technical solution of this disclosure embodiment is implemented as follows:

[0005] On one hand, this disclosure provides a log analysis method, including: in response to an analysis request for a work order to be analyzed, obtaining a trace identifier of the call chain that generated the work order to be analyzed; wherein the call chain is used to characterize the path of at least two layers of services called for the work order to be analyzed; based on the trace identifier, obtaining real-time log data and offline log data associated with the call chain from each layer of the service; and analyzing the real-time log data and the offline log data respectively to obtain the analysis result of the work order to be analyzed.

[0006] On the other hand, embodiments of this disclosure provide a log analysis apparatus, comprising: a first acquisition module, configured to acquire a trace identifier of the call chain that generated the work order to be analyzed in response to an analysis request for the work order to be analyzed; wherein the call chain is used to characterize the path of at least two layers of services called for the work order to be analyzed; a second acquisition module, configured to acquire real-time log data and offline log data associated with the call chain from each layer of the service based on the trace identifier; and an analysis module, configured to analyze the real-time log data and the offline log data respectively to obtain the analysis result of the work order to be analyzed.

[0007] In another aspect, embodiments of this disclosure provide a computer device including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0008] In another aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0009] In another aspect, embodiments of this disclosure provide a computer program including computer-readable code, which, when executed in a computer device, causes a processor in the computer device to perform some or all of the steps in the above-described method.

[0010] In another aspect, embodiments of this disclosure provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0011] Compared to related technologies, where the services called for a work order to be analyzed are independent, only log data from a single service can be obtained for analysis, this embodiment first obtains a trace identifier of the call chain that generated the work order in response to an analysis request. The call chain represents the path of at least two service layers called for the work order. This allows for accurate and rapid identification of all services called for the work order. Secondly, based on the trace identifier, real-time and offline log data associated with the call chain are obtained from each service layer. This allows for accurate and rapid acquisition of all types of log data associated with the work order. Finally, analyzing the real-time and offline log data separately yields simple and accurate analysis results for the work order, simplifying the log analysis process, reducing user workload, and improving the accuracy and efficiency of log analysis.

[0012] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0014] Figure 1 A schematic diagram illustrating the implementation flow of a log analysis method provided in this embodiment of the disclosure;

[0015] Figure 2 A schematic diagram illustrating the implementation flow of a log analysis method provided in this embodiment of the disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of a cloud-based work order log analysis and statistics system provided in an embodiment of this disclosure;

[0017] Figure 4 This is a schematic diagram of the composition structure of a log analysis device provided in an embodiment of the present disclosure;

[0018] Figure 5 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0022] This disclosure provides a log analysis method, which can be executed by a processor of a computer device. The computer device can refer to a server, laptop computer, tablet computer, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or any other device with log analysis capabilities. Figure 1 This is a schematic diagram illustrating the implementation flow of a log analysis method provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the method includes the following steps S101 to S103:

[0023] Step S101: In response to the analysis request for the work order to be analyzed, obtain the trace identifier of the call chain that generated the work order to be analyzed.

[0024] Here, a work order to be analyzed can refer to a work order or alarm that requires analysis to determine the cause, location, and other information of a fault. Work orders to be analyzed can include one or more types such as complaint work orders, fault work orders, problem work orders, requirement work orders, and change work orders. Work orders to be analyzed can be generated by front-end personnel; for example, complaint work orders are created by customer service personnel, and fault work orders are created by operations personnel. An analysis request can refer to a request used to determine the cause, location, and other information of a fault involved in a work order to be analyzed. The analysis request can carry an identifier for the work order to be analyzed. The identifier of the work order to be analyzed can be used to determine the work order to be analyzed corresponding to the current analysis request. There can be one or more corresponding work orders to be analyzed. During the implementation of step S101, in response to the user's selection operation for a work order to be analyzed, an analysis request matching the selection operation can be generated to respond to the analysis request for the work order to be analyzed.

[0025] In some embodiments, at least two service layers need to be invoked to generate work orders to be analyzed. These services can refer to software modules or components used to process user business, such as microservices. A microservice breaks down a project into multiple independent services, each of which can run independently and occupies a thread. Taking a distributed e-commerce system as an example, e-commerce software under a microservice architecture typically has an order microservice and a product microservice. The order microservice calls the product microservice to obtain information such as the product's identifier, name, price, and inventory to complete order-related business. Since generating work orders to be analyzed involves a large number of services, and these services may reside on different platforms or business systems and use different programming languages, the call chain for generating the work order can be determined to accurately identify all the services invoked.

[0026] Call chains can be used to characterize the path of at least two service layers called in response to a work order to be analyzed. These service layers can be nested. For example, the call chain for the first work order to be analyzed might be a call from the first service to the second service. If the second service experiences an internal failure, it will report an error, generating a first work order to be analyzed due to the second service failure. However, since the first service also calls the second service, it will also report an error. From the user's perspective, the error in the first service causes the business process to fail. Operations personnel see the error as the first service failing, and will submit a first work order to be analyzed for the first service failure so that developers can fix the problem. But in reality, the root cause of the first service error is a failure in the second service. Failure to accurately determine the call chain can lead to incorrect troubleshooting directions, resulting in slow processing, low efficiency, and high workload.

[0027] During the process of calling various service layers to generate a work order to be analyzed, a trace identifier (TraceID) can be added to determine the call chain. A trace identifier refers to the identifier of the call chain corresponding to a business request that generates a work order to be analyzed. In a single business request, a globally unique trace identifier is initially generated to identify this request. This trace identifier remains unchanged throughout the call process, regardless of how many service layers it passes through, and is continuously passed on with each service layer call. Ultimately, the entire path of the user's business request within the services can be linked together using the trace identifier. During the implementation of step S101, the correspondence between work orders to be analyzed and trace identifiers can be pre-stored. By determining the identifier of the work order to be analyzed carried in the analysis request, and based on this identifier and the pre-stored correspondence, the trace identifier of the call chain for the work order to be analyzed is determined.

[0028] Step S102: Based on the tracking identifier, obtain the real-time log data and offline log data associated with the call chain from each layer of the service.

[0029] Here, log data refers to process event records generated during system operation. Log data can be generated from the business side and undergo a series of cleaning and processing processes before analysis or feedback to the business. This process introduces a time window, which determines the timeliness of the log data. Based on data latency, log data can be categorized into offline log data and real-time log data; alternatively, based on the data collection method, the collected log data can be classified as real-time log data or offline log data matching the collection method. Collection methods can include real-time and offline log data collection. Real-time log data processing can be called stream processing, and offline log data processing can be called batch processing. During step S102, a trace identifier can be pre-appended to the generated real-time and offline log data to ensure that the working identifiers in the real-time and offline log data generated by all layer services in the same call chain are identical. Real-time and offline log data carrying the current trace identifier are then retrieved from the storage space containing all real-time and offline log data.

[0030] Step S103: Analyze the real-time log data and the offline log data respectively to obtain the analysis results of the work order to be analyzed.

[0031] Here, a first processing method can be used to analyze the real-time log data, yielding a first analysis result; a second processing method can be used to analyze the real-time log data, yielding a second analysis result. Integrating the first and second analysis results yields the analysis result for the work order to be analyzed. The first processing method is more efficient than the second. For example, real-time log data can be analyzed using access frequency analysis, while offline log data can be analyzed using characteristic character analysis. The first and second analysis results are then filtered and cleaned to obtain the analysis result for the work order to be analyzed, thereby determining the cause and location of the fault involved in the work order.

[0032] Compared to related technologies, where the services called for a work order to be analyzed are independent, only log data from a single service can be obtained for analysis, this embodiment first obtains a trace identifier of the call chain that generated the work order in response to an analysis request. The call chain represents the path of at least two service layers called for the work order. This allows for accurate and rapid identification of all services called for the work order. Secondly, based on the trace identifier, real-time and offline log data associated with the call chain are obtained from each service layer. This allows for accurate and rapid acquisition of all types of log data associated with the work order. Finally, analyzing the real-time and offline log data separately yields simple and accurate analysis results for the work order, simplifying the log analysis process, reducing user workload, and improving the accuracy and efficiency of log analysis.

[0033] In some embodiments, the above method may further include the following step S111 before implementing step S101:

[0034] Step S111: Using a preset class library and an aspect-oriented programming approach, add corresponding tracking identifiers to the log data generated in all layer services of the same call chain.

[0035] Here, a class library can refer to a packaged version of predefined class files, such as a Jar or War package. Class library files can be used not only for compression and distribution but also for deploying and encapsulating libraries, components, and plugins, and can be directly used by tools like compilers and Java Virtual Machines (JVMs). Aspect-Oriented Programming (AOP) is a technique that uses pre-compilation and runtime dynamic proxies to achieve unified maintenance of program functionality. AOP is a derivative paradigm of functional programming. AOP can isolate different parts of business logic, thereby reducing coupling between them, improving program reusability, and increasing development efficiency. Using predefined class libraries and AOP, corresponding trace identifiers are added to log data generated in all service layers within the same call chain, ensuring that the trace identifier for each call chain is globally unique. For example, by calling externally provided JAR files, different business systems on different types of cloud platforms can reference them; by appending global working identifiers to all log data using aspect-oriented programming, the tracing identifiers of the entire call chain are consistent. At the same time, JAR files are lightweight, low-intrusive, and transparent to applications, requiring only one configuration for business systems, thus reducing the burden on developers.

[0036] In this embodiment of the disclosure, by utilizing a preset class library and adopting aspect-oriented programming, corresponding tracking identifiers are added to the log data generated in all layer services of the same call chain. In this way, only one configuration is needed, so that even if new services or other changes are added later, the call chain does not need to be reconfigured and global tracking identifiers are added again.

[0037] In some embodiments, the above method may further include the following steps S121 to S123 before implementing step S101:

[0038] Step S121: Obtain initial log data of at least two log types from at least two layers of services of at least two of the business systems.

[0039] Here, the work order to be analyzed can involve at least two business systems. Business systems can refer to software modules or components used to execute user requests, such as a user management system, a tool management system, and a configuration management system. The user management system can be used for user login and managing user permissions, the tool management system can perform format conversions for various files and data, and the configuration management system can allow users to adjust computing resources. Each business system can have at least two service layers. For example, the user management system includes interface services and storage services, while the tool management system can include format conversion services and format recognition services. Initial log data can refer to the log data generated by each service itself. The type and storage location of each initial log data can be the same or different. The type of initial log data can be Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), OpenStack (cloud computing management platform), or Web type, etc. For example: obtain initial log data of IaaS and PaaS types from the first service, and initial log data of OpenStack and Web types from the second service, etc.

[0040] During the implementation of step S121, initial log data of at least two log types can be generated from at least two service layers of each of at least two business systems. The at least two service layers can be within the same business system or different business systems, and the initial log data of the at least two log types can be generated from the same service layer or different service layers.

[0041] Step S122: Using a preset adapter, the format of each initial log data is converted to obtain the format-converted initial log data.

[0042] Here, "adapter" can refer to an interface converter used to convert the data format of initial log data. Different types of initial log data can correspond to different data formats. The data volume of initial log data in a single data format can grow very large, becoming difficult to read and thus a performance bottleneck, requiring periodic cleanup. However, cleaning up a large initial log data file is a complex operation. Therefore, processing such as splitting the initial log data can be performed. Different initial log data formats can have different splitting formats; therefore, an adapter can be used to convert initial log data of different formats into initial log data of the same format.

[0043] Step S123: Use the log collection system to store the format-converted initial log data into a message queue.

[0044] Here, the initial log data after format conversion can be stored in a pre-defined message queue (Kafka) using a log collection system (Elasticsearch, Logstash, Kibana, ELK). For example, the real-time log data after format conversion can be stored in the first message queue, and the offline log data after format conversion can be stored in the second message queue, etc.

[0045] In some embodiments, step S102 may include step S124:

[0046] Step S124: Based on the tracking identifier, obtain the real-time log data and the offline log data associated with the call chain from the message queue.

[0047] Here, real-time log data and offline log data carrying the current tracking identifier can be obtained from the first message queue and the second message queue, respectively.

[0048] In this embodiment of the disclosure, by converting the format of initial log data of different log types obtained from each of at least two business systems and storing the converted initial log data in a message queue, it is helpful to simplify the processing flow and improve processing efficiency.

[0049] In some embodiments, step S124 above may include the following step S1241:

[0050] Step S1241: Based on the tracking identifier, obtain the real-time log data and the offline log data associated with the call chain from the message queue by calling a preset standardized interface.

[0051] Here, a standardized interface refers to an interface used to access message queues. This standardized interface can access different message queues in different ways, such as publish-subscribe, point-to-point, or request-response methods. During step S1241, message queues with different access methods can be pre-configured to store real-time and offline log data generated by different business systems or service layers. The pre-configured standardized interface is then used to retrieve real-time and offline log data carrying the current tracking identifier from these message queues.

[0052] Compared to calling multiple different interfaces to access different message queues, this embodiment of the disclosure uses a pre-defined standardized interface to obtain real-time log data and offline log data associated with the call chain from the message queue, which helps to improve processing efficiency.

[0053] This disclosure provides a log analysis method, such as... Figure 2 As shown, the method includes the following steps S201 to S205:

[0054] Steps S201 to S202 correspond to the aforementioned steps S101 to S102, and can be implemented with reference to the specific implementation of the aforementioned steps S101 to S102.

[0055] Step S203: The real-time log data is processed online using a distributed big data processing engine to obtain the first processing result.

[0056] Here, the distributed big data processing engine can refer to the open-source stream processing framework (Flink). Flink executes arbitrary streaming data programs in a data parallel and pipelined manner. Flink's pipeline runtime system can execute batch processing and stream processing programs, and can be used for online real-time computation of real-time log data to obtain the first processing result. For example, the first processing result indicates that the return value of the first called function in the first layer of service has an overflow.

[0057] Step S204: Use the cluster computing platform to perform offline processing on the first processing result and the offline log data to obtain the second processing result.

[0058] Here, the cluster computing platform can refer to a big data parallel computing framework (Spark) used for in-memory computing, which can be used to batch process the first processing result and offline log data to obtain a second processing result. For example, the second processing result indicates that the input value of the second called function in the second-layer service does not belong to the preset input format type.

[0059] Step S205: Perform online analysis on the first processing result and the second processing result to obtain the analysis result of the work order to be analyzed.

[0060] Online Analytical Processing (OLAP) is a software technology that enables analysts to quickly, consistently, and interactively observe information from various perspectives to achieve a deeper understanding of the data.

[0061] Compared to related technologies that process real-time log data online or offline log data separately and directly use the processing results as analysis results, this embodiment utilizes a distributed big data processing engine to process real-time log data online to obtain a first processing result. Then, a cluster computing platform is used to process both the first processing result and the offline log data offline to obtain a second processing result. Online analysis of both the first and second processing results helps to obtain more accurate analysis results for the work orders to be analyzed.

[0062] In some embodiments, the above method may further include the following steps S211 to S212 before implementing step S204:

[0063] Step S211: Determine the work order type of the work order to be analyzed.

[0064] Here, the work order type can include complaint work orders, fault work orders, problem work orders, request work orders, change work orders, etc. The format and structure of the work orders to be analyzed can differ for different work order types, and one or more attributes may vary. During the implementation of step S211, the correspondence between the identifier of the work order to be analyzed and the work order type can be pre-set. By determining the identifier of the work order to be analyzed, and based on the correspondence between the identifier and the work order type, the work order type corresponding to the identifier of the work order to be analyzed is determined.

[0065] Step S212: Obtain a work order template that matches the work order type.

[0066] Here, a work order template is used to record general information in a certain type of work order to be analyzed. Work order templates can include complaint work order templates, fault work order templates, problem work order templates, request work order templates, change work order templates, etc. Work order templates can be customized and edited by the user. During step S212, work order templates corresponding to different work order types can be preset, and all work order templates can be stored in a preset storage space. By determining the work order type of the work order to be analyzed, a work order template matching the current work order type can be retrieved from the preset storage space.

[0067] In some embodiments, step S204 above may include step S213 as follows:

[0068] Step S213: Based on the template information in the work order template that matches the work order type, the second processing result is obtained by calling the trained processing model in the cluster computing platform to extract data from the first processing result and the offline log data.

[0069] Here, the work order template carries general information (i.e., template information) for a certain type of work order to be analyzed. The trained processing model can refer to a pre-set machine learning model, such as a neural network model used to perform data extraction. During the implementation of step S213, the template information, the first processing result, and the offline log data can be input into the trained processing model to obtain the second processing result.

[0070] In this embodiment of the disclosure, by calling the trained processing model in the cluster computing platform to extract template information, first processing result and offline log data in the work order template, the second processing result can be accurately obtained, reducing workload and improving the accuracy of the second processing result.

[0071] In some embodiments, the above method may further include the following step S221 after implementing step S203:

[0072] Step S221: Store the first processing result in the data warehouse tool and the cache space respectively.

[0073] Here, the data warehouse tool (Hive) is set up based on the distributed system infrastructure (Hadoop) and is used for data extraction, transformation, and loading. It's a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The cache space (Remote Dictionary Server, Redis) is an open-source, network-enabled, in-memory or persistent log-structured key-value database written in a standard programming language (ANSI C), and provides interfaces for multiple languages. During step S221, the first processing result can be stored in the data warehouse tool first, and then stored in the cache space.

[0074] In some embodiments, step S205 may include step S222:

[0075] Step S222: In the data warehouse tool, perform online analysis on the first processing result and the second processing result to obtain the analysis result of the work order to be analyzed.

[0076] Alternatively, the second processing result can be stored in a data warehouse tool, where the first and second processing results can be analyzed online to obtain the analysis results of the work order to be analyzed.

[0077] In some embodiments, step S223 may be included after step S205:

[0078] Step S223: Store the analysis results in a relational database and the cache space.

[0079] Here, a relational database (MySQL) can store the analysis results in different tables, rather than storing them in a large repository, increasing read speed and improving flexibility. During step S223, the analysis results can first be stored in the relational database, and then stored in the cache space.

[0080] In some embodiments, after step S223 is implemented, in response to a user's query request, an ad-hoc query method can be used to obtain analysis results matching the query request from a relational database and cache space; and the analysis results can be displayed, etc.

[0081] In this embodiment of the disclosure, by storing the first processing result in a data warehouse tool and a cache space respectively, and performing online analysis on the first and second processing results in the data warehouse tool to obtain the analysis result of the work order to be analyzed, and storing the analysis result in a relational database and a cache space, it is helpful to accurately obtain the analysis result and improve the security of the analysis result transmission.

[0082] The log analysis method provided in this disclosure is illustrated below using a log analysis system determined based on the log analysis method as an example.

[0083] In this embodiment, addressing the shortcomings of cloud-related work order log analysis and statistics, and considering the characteristics of the work order system, the functions of the work order system are streamlined, several general modules are abstracted, and a mainstream distributed open-source framework is introduced to propose a cloud work order log analysis and statistics system. Figure 3 As shown, the system architecture design goal of this cloud work order log analysis and statistics system is a high-performance, highly available, scalable, extensible, secure, and agile distributed microservice system, which is divided into four layers from top to bottom: front-end presentation layer, application layer, service layer, and data layer.

[0084] The front-end presentation layer may include the Progressive Framework (Vue), the UI (ElementUI) component 301, the visualization and analysis charting (Echarts) component, the application programming interface (Rest Application Program Interface, Rest API) 302, etc.

[0085] The application layer core is based on a decentralized distributed task scheduling engine. It supports sharding and high availability of analysis tasks for pending work orders in distributed scenarios, enabling horizontal scaling of task throughput and execution efficiency. Task processing capacity elastically scales with resource availability. Appropriate resources are allocated to analysis tasks at appropriate times and activated. Identical analysis tasks are aggregated to the same executor for unified processing, and resources are dynamically allocated and added to newly allocated analysis tasks. The scheduling engine is thread-level scheduling, allowing tasks to be transparently integrated with different work order templates, achieving high decoupling and facilitating easier adaptation to the business development of work order systems on different cloud computing platforms.

[0086] The service layer, tailored to the characteristics of the cloud-based work order log analysis and statistics system, abstracts different modules to provide services to the task scheduling engine. The service layer mainly includes a user management module 303, a tool management module 304, a configuration management module 305, a log source module 306, a work order template module 307, a log collection module 308, a log analysis module 309, and a log application module 310, etc.

[0087] The user management module 303 includes user login and logout authentication functions, user access control functions, and user behavior log recording functions.

[0088] The tool management module 304 provides related tool services, including security protocols (Secure Shell, SSH), office software (Excel), and office software (Word), for importing and exporting log data. For example, users can choose to export log data in Word format, or import log data or work orders to be analyzed in batches into the system in a certain format.

[0089] The configuration management module 305 uses the Distributed Configuration Management Platform (Disconf) to configure data generated by common business processes, supporting hot deployment. The Distributed Configuration Management Platform focuses on common components / platforms for configuration management of various distributed systems, providing unified configuration management services. These services include server management and resource pool management. The user management module 303, tool management module 304, and configuration management module 305 can serve as the foundational platform for this system, used for pre-processing before log analysis.

[0090] The Log Source module 306 can use adapters to connect to different log sources; these sources include real-time and offline log data of various types such as IaaS, PaaS, SaaS, OpenStack, and Web. The adapters are used to convert the formats of different types of log data.

[0091] The work order template module 307 provides standardized interfaces for different cloud platforms and business systems to access and obtain log data. These different business systems include complaint work order systems, fault work order systems, and problem work order systems. The work order template module 307 also allows users to create and edit work order templates, including complaint work order templates, fault work order templates, problem work order templates, requirement work order templates, and change work order templates. In some embodiments, the work order template module 307 can be used to analyze and clean log data to generate work order analysis data. The work orders to be analyzed are created by front-end personnel. After the work orders to be analyzed are accessed by this system, the type of the current work order to be analyzed can be determined. A work order template for a certain type includes key information (i.e., template information) for that type of work order. For a certain type of work order to be analyzed, such as a complaint work order, the complaint work order template will be obtained. Then, based on the key information of the complaint work order template, the corresponding log content will be extracted from the log data, and the work order will be analyzed to obtain the analysis results.

[0092] The log collection module 308 can use the ELK system to send initial log data to the log analysis module via a message queue. The log collection module 308 can be divided into two parts: one part is the real-time collection of log data through the ELK system, and the other part is the offline batch processing of existing logs. The data integration here refers to the aggregation of log data after real-time and batch offline collection.

[0093] The log analysis module 309 can adopt a real-time and offline integrated computing solution, using Flink to perform real-time analysis on the incoming log data. Real-time analysis includes operation success rates and displays the first processing results of the overall statistical analysis in real time. These first processing results can then be either stored in a Hive library for offline analysis by Spark or written to Redis via Kafka. During offline analysis, the log analysis module 309 can call the Artificial Intelligence (AI) ticket service 311 to perform offline analysis on the offline log data to obtain second processing results; the AI ​​ticket service 311 can include a trained processing model. Finally, by online analysis and processing of the first and second processing results, the analysis results are obtained and stored in MySQL and Redis caches.

[0094] The log application module 310 may include functions such as work order analysis, work order display, and work order statistics.

[0095] In this embodiment, the cloud-based work order log analysis and statistics system integrates multiple existing mainstream open-source tool frameworks. Combining the characteristics of cloud computing services, it abstracts common components, making it open to extension and highly scalable. Users can easily configure analysis templates. Simultaneously, it integrates the collection of logs from all business operations, reducing pre-analysis preparation work, improving efficiency and accuracy, lowering the rate of human error, persisting raw log data and analysis results, providing multi-dimensional, multi-view real-time statistical displays, and supporting multi-user collaborative analysis.

[0096] Based on the foregoing embodiments, this disclosure provides a log analysis device, which includes various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0097] Figure 4 This is a schematic diagram of the composition structure of a log analysis device provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the log analysis device 400 includes: a first acquisition module 410, a second acquisition module 420, and an analysis module 430, wherein:

[0098] The first acquisition module 410 is used to acquire the trace identifier of the call chain that generated the work order to be analyzed in response to an analysis request for the work order to be analyzed; wherein the call chain is used to characterize the path of at least two layers of services called for the work order to be analyzed; the second acquisition module 420 is used to acquire real-time log data and offline log data associated with the call chain from each layer of the service based on the trace identifier; the analysis module 430 is used to analyze the real-time log data and the offline log data respectively to obtain the analysis result of the work order to be analyzed.

[0099] In some embodiments, the analysis module is further configured to: process the real-time log data online using a distributed big data processing engine to obtain a first processing result; process the first processing result and the offline log data offline using a cluster computing platform to obtain a second processing result; and perform online analysis processing on the first processing result and the second processing result to obtain the analysis result of the work order to be analyzed.

[0100] In some embodiments, the apparatus further includes: a determining module for determining the work order type of the work order to be analyzed; a third obtaining module for obtaining a work order template matching the work order type; and the analyzing module for: based on the template information in the work order template matching the work order type, extracting data from the first processing result and the offline log data by calling a trained processing model in the cluster computing platform to obtain the second processing result.

[0101] In some embodiments, the apparatus further includes: a first storage module, configured to store the first processing result in a data warehouse tool and a cache space respectively; the analysis module is further configured to: perform online analysis processing on the first processing result and the second processing result in the data warehouse tool to obtain the analysis result of the work order to be analyzed; the apparatus further includes: a second storage module, configured to store the analysis result in a relational database and the cache space.

[0102] In some embodiments, the apparatus further includes: an adding module, configured to use a preset class library and an aspect-oriented programming approach to add corresponding trace identifiers to log data generated in all layer services of the same call chain; wherein the trace identifier corresponding to each call chain is global and unique.

[0103] In some embodiments, the apparatus further includes: a fourth acquisition module, configured to acquire initial log data of at least two log types from at least two layers of services of at least two of the business systems; a conversion module, configured to perform format conversion on each of the initial log data using a preset adapter to obtain format-converted initial log data; a third storage module, configured to store the format-converted initial log data in a message queue using a log collection system; and the second acquisition module is further configured to: acquire the real-time log data and the offline log data associated with the call chain from the message queue based on the tracking identifier.

[0104] In some embodiments, the second acquisition module is further configured to: based on the tracking identifier, acquire the real-time log data and the offline log data associated with the call chain from the message queue by calling a preset standardized interface.

[0105] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0106] It should be noted that, in the embodiments of this disclosure, if the above-described log analysis method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0107] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0108] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0109] This disclosure provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0110] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0111] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0112] It should be noted that, Figure 5 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure, such as... Figure 5 As shown, the hardware entity of the computer device 500 includes: a processor 501, a communication interface 502, and a memory 503, wherein:

[0113] Processor 501 typically controls the overall operation of computer device 500.

[0114] Communication interface 502 enables computer devices to communicate with other terminals or servers via a network.

[0115] The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 501 and various modules in the computer device 500. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 501, the communication interface 502, and the memory 503 can be performed via bus 504.

[0116] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0117] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0118] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0119] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0121] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0122] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0123] The methods disclosed in the several method embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0124] If the embodiments of this disclosure involve personal information, the products using these embodiments have clearly informed the users of the personal information processing rules and obtained their voluntary consent before processing the personal information. If the embodiments of this disclosure involve sensitive personal information, the products using these embodiments have obtained the individual's separate consent before processing the sensitive personal information, and the requirement of "express consent" is also met.

[0125] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A log analysis method, characterized in that, include: In response to an analysis request for a work order to be analyzed, a trace identifier of the call chain that generated the work order to be analyzed is obtained; wherein, the call chain is used to characterize the path of at least two layers of services called for the work order to be analyzed; Based on the tracking identifier, real-time log data and offline log data associated with the call chain are obtained from each layer of the service. The real-time log data is processed online using a distributed big data processing engine to obtain a first processing result; Determine the work order type of the work order to be analyzed; Obtain a work order template that matches the work order type; Based on the template information in the work order template that matches the work order type, the second processing result is obtained by calling the trained processing model in the cluster computing platform to extract data from the first processing result and the offline log data. The first processing result and the second processing result are analyzed online to obtain the analysis result of the work order to be analyzed.

2. The method according to claim 1, characterized in that, The method further includes: The first processing result is stored in the data warehouse tool and the cache space, respectively. The online analysis of the first processing result and the second processing result to obtain the analysis result of the work order to be analyzed includes: In the data warehouse tool, the first processing result and the second processing result are analyzed online to obtain the analysis result of the work order to be analyzed; The method further includes: The analysis results are stored in a relational database and the cache space.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Using a pre-defined class library and employing aspect-oriented programming, corresponding trace identifiers are added to the log data generated in all service layers within the same call chain; wherein, the trace identifier corresponding to each call chain is global and unique.

4. The method according to claim 1, characterized in that, The method further includes: Obtain initial log data of at least two log types from at least two service layers of at least two business systems; Using a preset adapter, the format of each initial log data is converted to obtain the format-converted initial log data; The initial log data after format conversion is stored in a message queue using a log collection system. Based on the tracking identifier, the process of obtaining real-time log data and offline log data associated with the call chain from each layer of the service includes: Based on the tracking identifier, the real-time log data and the offline log data associated with the call chain are obtained from the message queue.

5. The method according to claim 4, characterized in that, The step of retrieving the real-time log data and the offline log data associated with the call chain from the message queue based on the tracking identifier includes: Based on the tracking identifier, the real-time log data and the offline log data associated with the call chain are obtained from the message queue by calling a preset standardized interface.

6. A log analysis device, characterized in that, include: The first acquisition module is used to acquire the trace identifier of the call chain that generated the work order to be analyzed in response to the analysis request; wherein, the call chain is used to characterize the path of at least two layers of services called for the work order to be analyzed. The second acquisition module is used to acquire, based on the tracking identifier, real-time log data and offline log data associated with the call chain from each layer of the service; The analysis module is used to process the real-time log data online using a distributed big data processing engine to obtain a first processing result; The determination module is used to determine the work order type of the work order to be analyzed; The third acquisition module is used to acquire a work order template that matches the work order type; The analysis module is further configured to extract data from the first processing result and the offline log data based on the template information in the work order template that matches the work order type, by calling the trained processing model in the cluster computing platform, to obtain a second processing result; and to perform online analysis processing on the first processing result and the second processing result to obtain the analysis result of the work order to be analyzed.

7. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5.

Images