A log processing method, apparatus, electronic device, and storage medium

Abstract

This application provides a log processing method, apparatus, electronic device, and storage medium. The method includes: receiving a target service query instruction; retrieving, based on the target service query instruction, historical target requests and historical target response results within a historical time period from the data storage unit of the log processing system; assembling the historical target requests according to the message assembly format of the original requests to obtain a target message; sending the target message to a target address to obtain the target response content fed back by the target address based on the target message; and obtaining a target analysis result based on the historical target response results and the target response content. This method improves the processing efficiency of log processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a log processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the growth of internet users, the volume of calls to IT system services has surged. During abnormal periods, it is necessary to locate the problem in abnormal requests, and during off-peak periods, to reproduce the problem.

[0003] Currently, the reproduction of requests is mainly done manually. This involves searching for the original request message in the log file or reconstructing the message according to the specifications, using tools to make calls to the service, observing the service logs or host logs, comparing the phenomena, and searching for errors from a large number of logs. In this way, the performance of the production system under test can be determined.

[0004] However, existing methods have the problem of poor detection results when inspecting production systems. Summary of the Invention

[0005] This application provides a log processing method, apparatus, electronic device, and storage medium to solve the problem that existing methods have poor detection results when detecting production systems.

[0006] Firstly, this application provides a log processing method, including:

[0007] Receive the target service query command;

[0008] Based on the target service query instruction, retrieve the target service's historical requests and historical response results within the historical time period from the data storage unit of the log processing system;

[0009] Based on the message assembly format of the original request, the target historical request is assembled to obtain the target message. The original request is the request initiated by the user against the production system that corresponds to the historical request. The production system is the system that corresponds to the target historical request and the target historical response result.

[0010] Send the target message to the target address to obtain the target response content fed back by the target address based on the target message. The target address is the address pre-configured by the user according to the target task.

[0011] Based on the historical response results and the content of the target response, the target analysis results are obtained.

[0012] In this application, the method further includes, before receiving the target service query instruction:

[0013] Obtain historical log information from the production system;

[0014] Based on the type of historical log information, the historical log information is classified to obtain classified log information;

[0015] Extract log information from the categorized log information, which includes historical requests and historical response results;

[0016] Based on the preset structured data requirements, the unstructured data in the target log information is processed into structured data.

[0017] Based on structured data, predictions for the operation of the production system are obtained;

[0018] Structured data and the corresponding prediction results are stored in the data storage unit so that the prediction results can be displayed in the display unit of the log processing system after the data storage unit receives a query request.

[0019] In this application, historical log information of the production system is obtained, including:

[0020] During the operation of the production system, the system obtains the operation information of the production system, the identification information of the service links, and the operation information of the middleware based on the log recording unit of the log processing system. The operation information includes request information, which includes the request, response header, message body, protocol, and execution time.

[0021] Historical log information is obtained based on the production system's operational information, service link identification information, and middleware's operational information.

[0022] In this application, structured data and the corresponding prediction results are stored in a data storage unit so that, after the data storage unit receives a query request, the prediction results are displayed in the display unit of the log processing system, including:

[0023] Based on the demand for data queries, structured data and the corresponding prediction results are divided into offline data and hot data.

[0024] Offline data is stored in a large data storage cluster within the data storage unit;

[0025] Hot data is stored in a distributed search engine within the data storage unit.

[0026] In this application, a target message is sent to a target address to obtain a target response from the target address based on the target message, including:

[0027] Determine the initial target address, which is the address pre-configured by the user based on the target task;

[0028] The initial target address configuration information is subjected to stress adjustment to obtain the target address. The configuration information includes the number of concurrent connections, the number of connections, and the target speed.

[0029] Send the target message to the target address to obtain the target response content from the target address based on the target message.

[0030] In this application, after sending the target message to the target address and receiving a response from the target address, the method further includes:

[0031] The response content is sent to the testing system. After the testing system obtains the offline test results based on the response content, it sends the offline test results to the result analysis unit of the log processing system to obtain the offline analysis results. The testing system is a system that simulates the operating status of the production system based on the response content.

[0032] The offline analysis results are displayed in the display unit of the log processing system.

[0033] In this application, based on historical response results and response content, the analysis results are obtained, including:

[0034] Obtain system indicator data corresponding to the response content;

[0035] Compare the differences between the response content and the execution result to obtain the difference analysis results;

[0036] Based on the system indicator data, the results of the error indicator data analysis are obtained;

[0037] The analysis results are obtained based on the results of the difference analysis and the error index data analysis.

[0038] In this application, after obtaining the analysis results based on the difference analysis results and the error index data analysis results, the method further includes:

[0039] The analysis results are sent to the display unit in the log processing system for replay display or system stress display.

[0040] In this application, after obtaining the analysis results based on historical response results and response content, the method further includes:

[0041] The analysis results are sent to the data storage unit for storage. After the data storage unit receives a query command for the analysis results, it sends the analysis results to the display unit in the log processing system for display.

[0042] Secondly, this application provides a log processing apparatus, comprising:

[0043] The receiving module is used to receive query commands for the target service;

[0044] The acquisition module is used to retrieve the target service's historical requests and historical response results within a historical time period from the data storage unit of the log processing system, based on the target service query instruction.

[0045] The message assembly module is used to assemble the target historical request according to the message assembly format of the original request to obtain the target message. The original request is the request initiated by the user to the production system and corresponding to the historical request. The production system is the system corresponding to the target historical request and the target historical response result.

[0046] The sending module is used to send the target message to the target address in order to obtain the target response content fed back by the target address based on the target message. The target address is the address pre-configured by the user according to the target task.

[0047] The module is used to obtain target analysis results based on the target's historical response results and target response content.

[0048] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0049] The memory stores the instructions that the computer executes;

[0050] The processor executes computer execution instructions stored in memory to implement the request processing method of this application.

[0051] Fourthly, a computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the request processing method of this application.

[0052] This application provides a log processing method, apparatus, electronic device, and storage medium. Upon receiving a target service query instruction, the method retrieves target historical requests and response results from the log processing system's data storage unit within a historical time period. Based on the original request's message assembly format, it assembles the target historical requests into a target message. The original request is a user-initiated request corresponding to a historical request to a production system. The production system corresponds to the target historical request and response results. The target message is then sent to a target address to obtain the target response content. The address is a pre-configured address by the user based on the target task. Then, based on the target historical response results and target response content, the method obtains the target analysis results. It can obtain the target historical requests and target historical response results of the production system within a historical time period according to the target service query command. By assembling the target historical request according to the message assembly format of the original request and sending the target message to the target address, the target historical requests of the production system within the historical time period can be accurately simulated, thereby obtaining the target response content. Finally, by analyzing the target historical response results and target response content, the target analysis results are obtained. This improves the problem of poor performance of existing methods. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 This is a schematic diagram of a log processing system provided in an embodiment of this application;

[0055] Figure 2 A flowchart illustrating a log processing method provided in an embodiment of this application;

[0056] Figure 3 A flowchart illustrating another log processing method provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the structure of the log processing device provided in the embodiments of this application;

[0058] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0059] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0060] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0061] In existing technologies, reproducing requests primarily involves manual processing. This includes searching log files for the original request message or reconstructing the message according to specifications, using tools to invoke the service, observing service logs or host logs, comparing the observed phenomena, searching for errors from a large volume of logs, and performing online performance monitoring and comparison with normal services. However, manually searching log files for the original request message is inefficient when dealing with a large volume of logs, and manually replaying requests does not reflect the actual load on the online system, making it impossible to reproduce the anomaly.

[0062] To address the aforementioned issues, this application employs a log processing system. Based on the target service query command, it automatically retrieves the target service's historical requests and responses within a historical timeframe from the log processing system's data storage unit. The system then assembles the target historical request into a target message according to the original request's message assembly format and sends it to a manually configured target address. This simulates the target historical requests within the production system over a historical timeframe, resulting in more accurate simulations and the acquisition of the target response content. Finally, by analyzing the target historical response results and content, the application obtains the target analysis results. Furthermore, directly retrieving the target service query command's corresponding historical requests and responses from the data storage unit is more efficient than manual target service queries, thus improving upon the shortcomings of existing methods in terms of performance.

[0063] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0064] Figure 1 This is a schematic diagram of a log processing system provided in an embodiment of this application, such as... Figure 1As shown, the log processing system includes: a production system, a testing system, a log acquisition unit, a log analysis unit, a data storage unit, a request control unit, a stress control unit, a result analysis unit, and a display unit. The log acquisition unit collects logs generated by the production system, cleans the collected logs through the log analysis unit, and outputs them to the data storage unit. The data storage unit stores the received data and responds to service status queries and result display requests from the request control unit. Upon receiving a request replay instruction, the request control unit queries the data storage unit for request replays of specified services within the corresponding time period. The stress control unit adjusts the stress on the requests issued by the request control unit to obtain the adjusted results. The request replay results and the adjusted results are output to the testing system for offline testing to obtain offline test results. After obtaining the offline test results, the offline test results, request replay results, and the adjusted results are output to the result analysis unit of the log processing system for analysis. The display unit stores or displays the analysis results. The execution subject of the log processing method provided in this application embodiment can be a log processing system. The log processing system can be a device such as a mobile phone, tablet, or computer. This embodiment does not impose any particular restrictions on the implementation method of the execution entity, as long as the execution entity can receive the target service query instruction; according to the target service query instruction, it obtains the target service's historical requests and historical response results within the historical time period from the data storage unit of the log processing system; according to the message assembly format of the original request, it assembles the target historical request into a target message, wherein the original request is the request initiated by the user against the production system corresponding to the historical request, and the production system is the system corresponding to the target historical request and the target historical response result; the target message is sent to the target address to obtain the target response content fed back by the target address based on the target message, wherein the target address is the address pre-configured by the user according to the target task; and the target analysis result is obtained based on the target historical response result and the target response content.

[0065] In this context, a production system refers to the system within an enterprise or organization used to produce and provide products or services. It typically includes multiple components such as hardware, software, networks, and databases, and is used to support the core business operations of the enterprise or organization.

[0066] A big data storage cluster is a computer cluster used to store and process large-scale data. It can consist of multiple servers and is used to efficiently store, manage, and retrieve large amounts of data. These clusters typically run a distributed file system (Hadoop Distributed File System) to ensure data reliability and scalability. Big data storage clusters can be used to support various big data processing tasks, including data analysis, data mining, machine learning, and more.

[0067] A distributed search engine is a system capable of performing efficient search and retrieval operations on large-scale datasets. It breaks down search tasks into multiple parts and distributes these parts across multiple computers (nodes) for simultaneous processing, thereby improving search speed and throughput. Distributed search engines typically use parallel computing and distributed storage to process query requests in order to achieve good performance when processing large amounts of data. Some common distributed search engines include Elasticsearch and Apache Solr.

[0068] Figure 2 This is a flowchart illustrating a log processing method provided in an embodiment of this application, as shown below. Figure 2 As shown, this log processing method includes the following steps:

[0069] S201, Receive the target service query command.

[0070] The target service query instruction can refer to an instruction to query a target service in the production system. In this embodiment, the target service query instruction can be an instruction sent by the user directly through the log processing system to request replay from the control unit, or it can be an instruction sent by the user through other system devices to request replay.

[0071] A target service can refer to a service that meets a specific goal or requirement. In this case, a target service can refer to a certain type of service within a target time period. In this embodiment of the application, when the production system executes a request, the data generated can carry a specific identifier, such as time or type of execution request. Therefore, the identifier can be queried through a target service query command.

[0072] In this embodiment of the application, before receiving the target service query instruction, the method further includes:

[0073] Obtain historical log information from the production system;

[0074] Based on the type of historical log information, the historical log information is classified to obtain classified log information;

[0075] Extract log information from the categorized log information, which includes historical requests and historical response results;

[0076] Based on the preset structured data requirements, the unstructured data in the target log information is processed into structured data.

[0077] Based on structured data, predictions for the operation of the production system are obtained;

[0078] Structured data and the corresponding prediction results are stored in the data storage unit so that the prediction results can be displayed in the display unit of the log processing system after the data storage unit receives a query request.

[0079] Historical log information can refer to logs generated during the historical operation of the production system. These logs may include service memory and CPU usage logs, as well as the memory, CPU, network, and disk usage logs of the host where the service is located.

[0080] Classifying historical log information can refer to methods of classifying historical log information according to different dimensions. These dimensions can include log level classification, functional module classification, time classification, source classification, business classification, and exception type classification.

[0081] Categorized log information refers to information obtained by classifying historical log information.

[0082] Extracting log information from categorized log information refers to the process of extracting historical requests and historical response results from categorized log information. Historical requests refer to requests that users have made to the production system, which contain the data or operations that the user needs. Historical response results refer to the processing results that the production system needs to process based on the data or operations in the historical requests and return to the user.

[0083] Structured data requirements refer to the requirements for processing unstructured data in target log information. Unstructured data refers to data in target log information that does not have a clearly defined data format or organization method. Since unstructured data is not suitable for storage and processing using traditional relational databases, it needs to be structured to obtain structured data.

[0084] Structured data is data organized in the form of tables, rows, and columns. Each data item has a clearly defined data type and definition. Structured data can be stored in a database and retrieved and manipulated using query languages ​​such as SQL. Typical structured data includes numbers, dates, and amounts. For example, tables in a relational database are a typical example of structured data.

[0085] Structured processing of unstructured data in target log information to obtain structured data can include text extraction methods. This involves using natural language processing techniques to extract key information, such as entities and keywords, from the text and mapping them to fields in a structured table for storage and analysis, thereby better utilizing different types of data. In this embodiment, after converting unstructured data into structured data, modeling the structured data can yield more accurate predictions of the future operation of the service.

[0086] In this embodiment of the application, obtaining historical log information of the production system includes:

[0087] During the operation of the production system, the system obtains the operation information of the production system, the identification information of the service links, and the operation information of the middleware based on the log recording unit of the log processing system. The operation information includes request information, which includes the request, response header, message body, protocol, and execution time.

[0088] Historical log information is obtained based on the production system's operational information, service link identification information, and middleware's operational information.

[0089] Among them, the operational information of the production system refers to the data and information about the actual production system in operation. This operational information can include performance indicators, such as response time, throughput, and concurrent connections, used to measure the system's performance and efficiency; resource utilization, involving the system's resource usage, such as CPU utilization, memory usage, and disk space; errors and anomalies, recording errors, anomalies, and warnings generated by the system to help troubleshoot and resolve problems; log recording, including various log information generated by the system; user behavior, information about user operations, activities, and access within the system; data traffic, for network systems, recording information such as data traffic, transmission rate, and bandwidth utilization; and status monitoring, monitoring various system states, such as operating status, downtime, and availability.

[0090] Service chain identification information is used to uniquely identify and recognize the relationships and processes between services in a distributed system or microservice architecture. Service chain identifiers are typically used to trace the path a request takes through the system, enabling the monitoring and analysis of performance, latency, errors, and efficiency metrics throughout the service chain. This identification information typically includes the following: Trace ID, Span ID, parent-child relationship, timestamp, and additional information recorded and transmitted. This identification information achieves the following goals: tracing distributed requests, error tracing, monitoring, and diagnosis. Through service chain identification information, we can better understand how a distributed system works and how requests are passed and processed between different components.

[0091] Middleware runtime information refers to the status, performance metrics, and operation records of middleware used in distributed systems, applications, or services. Middleware is a software layer located between the operating system and applications, handling fundamental functions such as communication, data transmission, and transaction management. It helps different components or services collaborate and plays a crucial role in ensuring the stability, performance, and security of distributed systems.

[0092] In this embodiment, structured data and corresponding prediction results are stored in a data storage unit so that the prediction results are displayed in the display unit of the log processing system after the data storage unit receives a query request. This includes:

[0093] Based on the demand for data queries, structured data and the corresponding prediction results are divided into offline data and hot data.

[0094] Offline data is stored in a large data storage cluster within the data storage unit;

[0095] Hot data is stored in a distributed search engine within the data storage unit.

[0096] Among them, the demand for data query popularity can refer to the demand for querying and analyzing the popularity of data or information. This demand for data query popularity can be determined based on the query behavior of users recorded by the system. For example, based on the frequency of user queries for target category data recorded by the system, different categories of structured data and the prediction results corresponding to the structured data can be divided into offline data and hot data.

[0097] Offline data can refer to data that does not meet the demand for high-frequency data queries. Since the query frequency of this data is low and the response requirements are also low, there is no need to deal with high-frequency query situations. Therefore, it can be stored in the big data storage cluster in the data storage unit.

[0098] Hot data refers to data that meets the demand for high-frequency data queries. Because this data is queried frequently, the requirements for response are also high. It is necessary to deal with high-frequency query situations. Therefore, it can be stored in the distributed search engine in the data storage unit to improve the query response rate of hot data through the distributed search engine.

[0099] S202. Based on the target service query instruction, retrieve the target service's historical requests and historical response results within the historical time period from the data storage unit of the log processing system.

[0100] Among them, the target historical request can refer to the request determined from the historical requests according to the target service query instruction, and the target historical response result is the response result of the target historical request.

[0101] In this embodiment of the application, the process of obtaining the target historical requests and target historical response results of the target service within a historical time period from the data storage unit of the log processing system according to the target service query instruction can be as follows: Based on the target service query instruction, determine the historical requests and corresponding historical response results of the target service query instruction in the data storage unit, and obtain the target historical requests and target historical response results of the target service within a historical time period from the data storage unit of the log processing system.

[0102] S203. According to the message assembly format of the original request, the target historical request is assembled into a message to obtain the target message. The original request is the request initiated by the user to the production system and corresponding to the historical request. The production system is the system corresponding to the target historical request and the target historical response result.

[0103] The original request can refer to the request sent by the user to the current production system.

[0104] Message assembly refers to the process of combining data into a complete message or message according to a specific format and rules. In the embodiments of this application, the target historical request is assembled into the original message according to the order of the original request to obtain the target message.

[0105] Messages can be used to fulfill requests. A message is a data unit transmitted in a computer network that contains the content of the request and related metadata, such as the request method, request headers, and request body.

[0106] S204. Send the target message to the target address to obtain the target response content fed back by the target address based on the target message, wherein the target address is the address pre-configured by the user according to the target task.

[0107] In network communication, the destination address is an identifier used to identify which device or host a data packet should be sent to. The type and format of the destination address vary across different communication protocols. For example, in Ethernet, the destination address is typically a MAC address, while in the Internet Protocol (IP) protocol, it is an IP address. This address helps ensure that data packets are correctly delivered to the target device. In this embodiment, the destination address can be an address pre-configured by the user according to the target task.

[0108] The target response content refers to the feedback response expected or desired from the receiver in a given context. This response is usually made in response to previous actions, information transmissions, or requests, and is used to confirm receipt, provide results, or answer questions. The target response content varies depending on the context and can be information in different forms such as text, data, sound, and images. This response content is a response to the target message and includes information about whether the operation was successful, the result of the request, and the required information. In the embodiments of this application, the target response content fed back based on the target message refers to the response content fed back by the receiver based on the target address of the target message sent.

[0109] In this embodiment of the application, the method for sending a target message to a target address to obtain the target response content fed back by the target address based on the target message may include:

[0110] Determine the initial target address, which is the address pre-configured by the user based on the target task;

[0111] The initial target address configuration information is subjected to stress adjustment to obtain the target address. The configuration information includes the number of concurrent connections, the number of connections, and the target speed.

[0112] Send the target message to the target address to obtain the target response content from the target address based on the target message.

[0113] The stress adjustment process refers to dynamically adjusting the resource allocation of the production system based on its load during operation to ensure stability and performance. In this embodiment, stress adjustment can be performed on the initial target address configuration information by configuring the number of concurrent connections, the number of connections, and the target rate.

[0114] Concurrency refers to the number of requests processed simultaneously. In stress testing, the performance limits of a production system can be tested by gradually increasing the number of concurrent requests. Increasing concurrency increases the load on the production system; if the system cannot handle these concurrent requests, performance bottlenecks or system crashes will occur. Therefore, during stress testing, it is necessary to gradually increase the number of concurrent requests and observe metrics such as system response time and throughput to determine the performance limits and bottlenecks of the production system. Simultaneously, the number of concurrent requests should be set reasonably according to actual conditions to ensure the stability and reliability of the production system.

[0115] The number of links can refer to the number of connections established simultaneously. By configuring the number of links to the target address, a corresponding number of links are created to apply connection pressure to the target address.

[0116] The target rate can refer to the number of requests or the amount of data that the production system needs to process within a certain period of time. By configuring the target rate mode, the corresponding traffic rate management algorithm is used to manage the flow rate so as to achieve the target address to generate the corresponding system throughput within a fixed time period.

[0117] In this embodiment of the application, after sending the target message to the target address to obtain the response content from the target address, the method further includes:

[0118] The response content is sent to the testing system. After the testing system obtains the offline test results based on the response content, it sends the offline test results to the result analysis unit of the log processing system to obtain the offline analysis results. The testing system is a system that simulates the operating status of the production system based on the response content.

[0119] The offline analysis results are displayed in the display unit of the log processing system.

[0120] The testing system can refer to a system identical to the production system, used to simulate the operation of the production system. In this embodiment, the testing system can receive response content and perform offline testing based on the response content, thereby obtaining offline test results. After obtaining the offline test results, the offline test results are sent to the result analysis unit of the log processing system to obtain offline analysis results, which are then displayed through the display unit.

[0121] S205. Based on the historical response results and the content of the target response, the target analysis results are obtained.

[0122] The target analysis results can refer to the analysis results of the target's historical response results and target response content. These results characterize the operational status of the production system. In this embodiment, by comparing the target's historical response results and target response content, including execution time, comparative analysis results characterizing similarity and differences from existing content can be obtained. Simultaneously, comparative analysis results characterizing other indicators can be obtained by comparing service memory and CPU utilization, and host memory and CPU utilization. Furthermore, by marking desired error messages, it is possible to determine which requests will generate marked errors, thereby obtaining comparative analysis results characterizing production problems. Through these comparative analysis results, the target analysis results can be obtained.

[0123] In this embodiment of the application, the method for obtaining the analysis results based on historical response results and target response content may include:

[0124] Obtain system indicator data corresponding to the response content;

[0125] Compare the differences between the response content and the execution result to obtain the difference analysis results;

[0126] Based on the system indicator data, the results of the error indicator data analysis are obtained;

[0127] The analysis results are obtained based on the results of the difference analysis and the error index data analysis.

[0128] Difference analysis is a method used to compare the differences or similarities between two or more objects, datasets, samples, etc. Difference analysis can better determine the relationships, trends, and patterns between data. In this embodiment, the difference analysis results are obtained by comparing the difference between the response content and the execution result.

[0129] In this embodiment of the application, after obtaining the analysis results based on the difference analysis results and the error index data analysis results, the method further includes:

[0130] The analysis results are sent to the display unit in the log processing system for replay display or system stress display.

[0131] Among them, replay result display refers to the process of requesting replay and displaying the results after obtaining the analysis results, in order to reproduce specific problems or scenarios.

[0132] System pressure display refers to the process of displaying the pressure status after a system pressure regulation simulation is performed.

[0133] In this embodiment of the application, after obtaining the analysis results based on historical response results and response content, the method further includes:

[0134] The analysis results are sent to the data storage unit for storage. After the data storage unit receives a query command for the analysis results, it sends the analysis results to the display unit in the log processing system for display.

[0135] The log processing method provided in this application can automatically obtain the target historical requests and target historical response results of the production system within a historical time period according to the target service query instruction. By assembling the target historical requests according to the message assembly format of the original requests and sending the target message to the target address, the target historical requests of the production system within the historical time period can be accurately simulated to achieve the real pressure of the online system, thereby obtaining the target response content. Finally, by analyzing the target historical response results and target response content, the target analysis results are obtained. Thus, the problem of poor performance of existing methods is improved.

[0136] Figure 3 A flowchart illustrating another log processing method provided in this application embodiment is shown below. Figure 3 As shown, the log processing method includes:

[0137] S301. Obtain historical log information generated during the operation of the production system;

[0138] S302. Filter and classify different types of logs and store them in the data storage unit.

[0139] The data storage unit is used to receive data output by the persistent log analysis module and respond to the query service status. It categorizes the data query requests. For queries of hot data, the data is cached from the distributed search engine for easy and quick re-query. For queries of large historical data volumes, an offline processing method is used to issue a data query task, and the historical data is categorized in the background. After the query is completed, a notification is issued.

[0140] S303. Receive a request to replay, query the requested data in the data storage unit within the corresponding time period, and send a replay request to the specified service.

[0141] Specifically, based on the requests filtered after the query, the request message is assembled according to the original request message format, and the request is initiated according to the request target address configured by the user.

[0142] S304. Output the result of the replay request to the result analysis unit for analysis.

[0143] Specifically, based on the configured number of concurrent requests to the target address, corresponding concurrent request threads are created, and the initiated requests are sequentially allocated to different task queues; based on the configured number of connections to the target address, a corresponding number of connections are created to achieve connection pressure on the target address; and according to the configured target rate mode, a corresponding traffic rate management algorithm is used for flow rate management to achieve the target address generating the corresponding system throughput within a fixed time period.

[0144] In this embodiment of the application, between steps S303 and S304, pressure adjustment can be performed based on the replay request issued to the specified service, and the replay of the specified service can be performed based on the result of the pressure adjustment.

[0145] Results analysis includes: comparing the difference between the response content and the execution result, comparing service memory and CPU usage, comparing host memory and CPU usage, and obtaining the difference analysis results; marking the error messages that are expected to appear, and obtaining the error indicator data analysis results based on system indicator data.

[0146] S305. Input the analyzed results into a database or output them for display.

[0147] This application provides another log processing method that solves the problem of low efficiency in manually retrieving logs. It also improves upon existing methods that typically use rolling storage for log files, leading to file loss over time. Furthermore, it addresses the issue that manually replaying requests may not accurately reflect the real-world pressure of the online system, making it difficult to reproduce anomalies. Additionally, it addresses the limitations of using scripts with load testing tools to automatically filter request messages for desired time periods and accurately simulate boundary conditions that trigger anomalies. By recording service call logs generated during production system runtime, analyzing and storing these logs, and replaying requests within a selected time period at appropriate times, this method simulates real online pressure using a pressure adjustment module and analyzes the results of the request replay using a result analysis module and a service status monitoring module. This allows for performance tuning of online services or problem reproduction, thus improving the effectiveness of log processing.

[0148] Figure 4 This is a schematic diagram of the log processing device provided in an embodiment of this application. Figure 4 As shown, the log processing device 40 includes: a receiving module 401, an acquisition module 402, a message assembly module 403, a sending module 404, and an obtaining module 405. Wherein:

[0149] Receiver module 401 is used to receive target service query instructions;

[0150] The acquisition module 402 is used to acquire the target historical requests and target historical response results of the target service within a historical time period from the data storage unit of the log processing system according to the target service query instruction.

[0151] The message assembly module 403 is used to assemble the target historical request according to the message assembly format of the original request to obtain the target message. The original request is the request initiated by the user to the production system and corresponding to the historical request. The production system is the system corresponding to the target historical request and the target historical response result.

[0152] The sending module 404 is used to send the target message to the target address in order to obtain the target response content fed back by the target address based on the target message, wherein the target address is the address pre-configured by the user according to the target task;

[0153] Module 405 is used to obtain target analysis results based on the target's historical response results and target response content.

[0154] In this embodiment of the application, the acquisition module 402 can also be used for:

[0155] Obtain historical log information from the production system;

[0156] Based on the type of historical log information, the historical log information is classified to obtain classified log information;

[0157] Extract log information from the categorized log information, which includes historical requests and historical response results;

[0158] Based on the preset structured data requirements, the unstructured data in the target log information is processed into structured data.

[0159] Based on structured data, predictions for the operation of the production system are obtained;

[0160] Structured data and the corresponding prediction results are stored in the data storage unit so that the prediction results can be displayed in the display unit of the log processing system after the data storage unit receives a query request.

[0161] In this embodiment of the application, the acquisition module 402 can also be specifically used for:

[0162] During the operation of the production system, the system obtains the operation information of the production system, the identification information of the service links, and the operation information of the middleware based on the log recording unit of the log processing system. The operation information includes request information, which includes the request, response header, message body, protocol, and execution time.

[0163] Historical log information is obtained based on the production system's operational information, service link identification information, and middleware's operational information.

[0164] In this embodiment of the application, the acquisition module 402 can also be specifically used for:

[0165] Based on the demand for data queries, structured data and the corresponding prediction results are divided into offline data and hot data.

[0166] Offline data is stored in a large data storage cluster within the data storage unit;

[0167] Hot data is stored in a distributed search engine within the data storage unit.

[0168] In this embodiment of the application, the sending module 404 can also be used for:

[0169] Determine the initial target address, which is the address pre-configured by the user based on the target task;

[0170] The initial target address configuration information is subjected to stress adjustment to obtain the target address. The configuration information includes the number of concurrent connections, the number of connections, and the target speed.

[0171] Send the target message to the target address to obtain the target response content from the target address based on the target message.

[0172] In this embodiment of the application, module 405 can also be used for:

[0173] Obtain system indicator data corresponding to the response content;

[0174] Compare the differences between the response content and the execution result to obtain the difference analysis results;

[0175] Based on the system indicator data, the results of the error indicator data analysis are obtained;

[0176] The analysis results are obtained based on the results of the difference analysis and the error index data analysis.

[0177] In this embodiment of the application, module 405 can also be used for:

[0178] The analysis results are sent to the display unit in the log processing system for replay display or system stress display.

[0179] In this embodiment of the application, module 405 can also be used for

[0180] The analysis results are sent to the data storage unit for storage. After the data storage unit receives a query command for the analysis results, it sends the analysis results to the display unit in the log processing system for display.

[0181] As can be seen from the above, the log processing device in this embodiment comprises a receiving module 401 for receiving a target service query instruction; an acquisition module 402 for acquiring, according to the target service query instruction, the target historical requests and target historical response results of the target service within a historical time period from the data storage unit of the log processing system; a message assembly module 403 for assembling the target historical request according to the message assembly format of the original request to obtain a target message, wherein the original request is a request initiated by the user against the production system corresponding to the historical request, and the production system is the system corresponding to the target historical request and the target historical response result; a sending module 404 for sending the target message to the target address to obtain the target response content fed back by the target address based on the target message, wherein the target address is an address pre-configured by the user according to the target task; and an acquisition module 405 for obtaining the target analysis result based on the target historical response result and the target response content. This improves the efficiency of log processing.

[0182] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 includes:

[0183] The electronic device 50 may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a communication component 503, and other components. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0184] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the log processing method described above.

[0185] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0186] In the above Figure 5In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0187] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0188] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0189] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described log processing methods.

[0190] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0191] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0192] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the log processing methods provided in embodiments of this application.

[0193] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0194] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0195] Since the instructions stored in the storage medium can execute the steps in any of the log processing methods provided in the embodiments of this application, the beneficial effects that any of the log processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0196] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0197] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A log processing method characterized by, Applied to a log processing system, the method includes: Receive the target service query command; According to the target service query instruction, the target service’s target historical requests and target historical response results within the historical time period are obtained from the data storage unit of the log processing system. According to the message assembly format of the original request, the target historical request is assembled to obtain the target message. The original request is a request initiated by the user to the production system that corresponds to the historical request. The production system is the system that corresponds to the target historical request and the target historical response result. The target message is sent to the target address to obtain the target response content fed back by the target address based on the target message, wherein the target address is an address pre-configured by the user according to the target task; The step of sending the target message to the target address to obtain the target response content fed back by the target address based on the target message includes: Determine the initial target address, which is the address pre-configured by the user according to the target task; The initial target address is subjected to stress adjustment processing based on its configuration information to obtain the target address. The configuration information includes the number of concurrent connections, the number of connections, and the target speed. The target message is sent to the target address to obtain the target response content returned by the target address based on the target message; Based on the target's historical response results and the target's response content, the target analysis results are obtained; The step of obtaining target analysis results based on the target's historical response results and the target's response content includes: Obtain system indicator data corresponding to the response content; By comparing the difference between the response content and the execution result, the difference analysis results are obtained. Based on the system indicator data, the error indicator data analysis results are obtained; The analysis results are obtained based on the difference analysis results and the error index data analysis results.

2. The method of claim 1, wherein, Before receiving the target service query instruction, the method further includes: Obtain the historical log information of the production system; Based on the type of the historical log information, the historical log information is classified to obtain classified log information; Extract log information from the categorized log information, the log information including historical requests and historical response results; Based on the preset structured data requirements, the unstructured data in the target log information is processed into structured data. Based on the structured data, the predicted results of the production system operation are obtained; The structured data and the prediction results corresponding to the structured data are stored in the data storage unit so that the prediction results are displayed in the display unit of the log processing system after the data storage unit receives a query request.

3. The method according to claim 2, characterized in that, The acquisition of historical log information from the production system includes: When the production system is running, the operation information of the production system, the identification information of the service link, and the operation information of the middleware are obtained according to the log recording unit of the log processing system. The operation information includes request information, which includes a request, response header, message body, protocol, and execution time. Historical log information is obtained based on the operation information of the production system, the identification information of the service links, and the operation information of the middleware.

4. The method according to claim 2, characterized in that, The step of storing the structured data and the prediction results corresponding to the structured data in the data storage unit, so as to display the prediction results in the display unit of the log processing system after the data storage unit receives a query request, includes: Based on the data query popularity demand, the structured data and the prediction results corresponding to the structured data are divided to obtain offline data and hot data; The offline data is stored in the big data storage cluster in the data storage unit; The hot data is stored in the distributed search engine within the data storage unit.

5. The method according to claim 1, characterized in that, After sending the target message to the target address to obtain the response content returned by the target address, the method further includes: The response content is sent to the testing system, and after the testing system obtains the offline test results based on the response content, the offline test results are sent to the result analysis unit of the log processing system to obtain offline analysis results. The testing system is a system that simulates the operating status of the production system based on the response content. The offline analysis results are displayed in the display unit of the log processing system.

6. The method according to claim 1, characterized in that, After obtaining the analysis results based on the difference analysis results and the error index data analysis results, the method further includes: The analysis results are sent to the display unit of the log processing system for replay display or system stress display.

7. The method according to claim 1, characterized in that, After obtaining the analysis results based on the historical response results and the response content, the method further includes: The analysis results are sent to the data storage unit for storage. After the data storage unit receives a query instruction for the analysis results, it sends the analysis results to the display unit in the log processing system for display.

8. A log processing device, characterized in that, include: The receiving module is used to receive query commands for the target service; The acquisition module is used to acquire the target historical requests and target historical response results of the target service within a historical time period from the data storage unit of the log processing system according to the target service query instruction. The message assembly module is used to assemble the target historical request according to the message assembly format of the original request to obtain the target message, wherein the original request is a request initiated by the user to the production system and corresponding to the historical request, and the production system is the system corresponding to the target historical request and the target historical response result; A sending module is used to send the target message to a target address to obtain the target response content fed back by the target address based on the target message, wherein the target address is an address pre-configured by the user according to the target task; The module is used to obtain target analysis results based on the target's historical response results and the target's response content.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the request processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the request processing method as described in any one of claims 1 to 7.

Images