Log processing method and device and electronic equipment

Abstract

The embodiment of the invention provides a log processing method and device and electronic equipment, and relates to the technical field of big data. According to the method, a predefined log format is generated in response to an execution request of a current test case, multiple pieces of server information associated with the current test case are obtained, after a test is triggered, a unique service identifier returned by a tested system is obtained, a matching rule is constructed according to the unique service identifier and the predefined log format, and the server information is matched according to the matching rule. According to the method, the log records associated with the current test case are screened out from the log files of the multiple servers, the problems that logs are scattered and manual positioning is difficult in a distributed environment are solved, and the accuracy of log association and the collection efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a log processing method and device and electronic equipment. BACKGROUND

[0002] In the field of software development and testing, especially in industries such as finance and the Internet that have extremely high requirements for system stability, the execution path of business logic needs to be tracked during the development and testing of programs. For example, in a bank core system or an online payment platform, a business operation usually involves the cooperative work of multiple server components, and each component generates a corresponding log file during execution. These log files record the running state of the system and the business processing process, and are an important basis for verifying whether the business flow is executed according to the expected path. Developers or testers analyze log files to ensure the correctness and stability of the system.

[0003] In the prior art, manual operation is mainly relied on to complete log acquisition and analysis. Testers need to manually log in to multiple servers and filter out log clues related to the current test case from massive log data. The time stamps and business numbers in the logs of different servers are manually compared to determine whether the logic of program execution meets expectations.

[0004] Although the method of the prior art can meet the basic needs, in a large-scale distributed system, as the business complexity and the number of servers increase, manual operation cannot meet the efficient and accurate log analysis needs. SUMMARY

[0005] The present application provides a log processing method, device and electronic equipment to solve the technical defects of low efficiency and poor accuracy in the prior art when manually acquiring log records and comparing log processes.

[0006] In a first aspect, the present application provides a log processing method, comprising:

[0007] In response to an execution request of a current test case, a predefined log format is generated, the predefined log format including a fixed prefix and a field separation structure;

[0008] A plurality of server information associated with the current test case is acquired, the server information being used to indicate a log collection source, and the server information including a server IP address, a log path and a time stamp synchronization rule;

[0009] After triggering the test, a unique business identifier returned by the tested system is acquired;

[0010] According to the unique service identifier and the predefined log format, a matching rule is constructed to filter log records associated with the current test case from log files of the plurality of servers.

[0011] Optionally, the predefined log format is generated in response to the execution request of the current test case, including:

[0012] In response to the execution request of the current test case, a service operation type corresponding to the execution request is determined.

[0013] Based on the service operation type, a log template corresponding to the service operation type is obtained from a preset rule library, the log template defining the fixed prefix and the field separation structure.

[0014] According to the log template, the predefined log format is generated.

[0015] Optionally, the matching rule is constructed according to the unique service identifier and the predefined log format to filter log records associated with the current test case from log files of the plurality of servers, including:

[0016] Based on the server IP address and the log path, log files are obtained from the plurality of servers.

[0017] Based on the fixed prefix and the field separation structure in the predefined log format, a matching pattern is generated.

[0018] According to the matching pattern and the unique service identifier, a regular expression is generated.

[0019] Based on the regular expression, the log files are subjected to matching processing.

[0020] Log records matched successfully are taken as log records associated with the current test case.

[0021] Optionally, the log files are subjected to matching processing based on the regular expression, including:

[0022] The regular expression is compiled into a regular matching object.

[0023] Log files of each of the servers are read line by line, and the regular matching object is invoked to perform a matching operation on current line log text.

[0024] If the matching operation returns success, the current line log text is retained.

[0025] If the matching operation returns failure, the current line log text is skipped.

[0026] Optionally, the method further includes:

[0027] extracting a raw timestamp of the filtered log record and an associated server identifier;

[0028] obtaining a timestamp synchronization rule of the server according to the server identifier, and correcting the raw timestamp based on the timestamp synchronization rule to obtain a corrected timestamp;

[0029] sorting the log record according to the corrected timestamp to obtain an ordered log sequence.

[0030] Optionally, the method further comprises:

[0031] obtaining a predefined log procedure rule associated with the current test case, the predefined log procedure rule comprising a plurality of log keywords arranged in an expected execution order;

[0032] sequentially searching for the plurality of log keywords in the ordered log sequence in time order;

[0033] determining whether the plurality of log keywords are all found;

[0034] if so, determining whether the order is consistent with the expected execution order of the predefined log procedure rule;

[0035] in the case where the order is consistent with the expected execution order of the predefined log procedure rule, determining that an actual execution path of the program under test conforms to the expectation.

[0036] Optionally, before the generating the predefined log format according to the log template, the method further comprises:

[0037] counting a triggering frequency of each business operation type within a preset period of time;

[0038] determining a business operation type whose triggering frequency exceeds a preset frequency threshold as a high-frequency business operation type;

[0039] caching a log template corresponding to the high-frequency business operation type to a local memory;

[0040] when the business operation type is a high-frequency business operation type, reading the log template from the local memory for generating the predefined log format.

[0041] In a second aspect, the present application provides a log processing apparatus, comprising:

[0042] a processing module configured to generate a predefined log format in response to an execution request of a current test case, the predefined log format comprising a fixed prefix and a field separation structure;

[0043] The acquisition module is configured to acquire a plurality of server information associated with a current test case, the server information being used to indicate a log collection source, and the server information including a server IP address, a log path, and a timestamp synchronization rule.

[0044] The acquisition module is further configured to acquire a unique service identifier returned by a tested system after triggering a test.

[0045] The processing module is further configured to construct a matching rule according to the unique service identifier and the predefined log format, and filter log records associated with the current test case from log files of the plurality of servers.

[0046] Optionally, the apparatus further includes a determination module.

[0047] The determination module is configured to determine a service operation type corresponding to an execution request of a current test case in response to the execution request.

[0048] The acquisition module is further configured to acquire a corresponding log template from a preset rule library based on the service operation type, the log template defining the fixed prefix and the field separation structure.

[0049] The processing module is further configured to generate the predefined log format according to the log template.

[0050] Optionally, the acquisition module is further configured to acquire log files from the plurality of servers based on the server IP address and the log path.

[0051] The processing module is further configured to generate a matching pattern based on the fixed prefix and the field separation structure in the predefined log format.

[0052] The processing module is further configured to generate a regular expression according to the matching pattern and the unique service identifier.

[0053] The processing module is further configured to perform matching processing on the log files based on the regular expression.

[0054] The processing module is further configured to take a log that is matched successfully as a log record associated with the current test case.

[0055] Optionally, the processing module is further configured to compile the regular expression into a regular matching object.

[0056] The processing module is further configured to read log files of each of the servers line by line, and invoke the regular matching object to perform a matching operation on a current line of log text.

[0057] The processing module is further configured to, if the matching operation returns success, keep the current line of log text.

[0058] The processing module is further configured to, if the matching operation returns failure, skip the current line of log text.

[0059] Optionally, the processing module is further configured to extract an original timestamp of the screened log record and an associated server identifier.

[0060] The obtaining module is further configured to obtain a timestamp synchronization rule of the server according to the server identifier.

[0061] The processing module is further configured to correct the original timestamp based on the timestamp synchronization rule to obtain a corrected timestamp.

[0062] The processing module is further configured to sort the log record according to the corrected timestamp to obtain an ordered log sequence.

[0063] Optionally, the apparatus further comprises a judging module.

[0064] The obtaining module is further configured to obtain a predefined log procedure rule associated with the current test case, the predefined log procedure rule comprising a plurality of log keywords arranged in an expected execution order.

[0065] The processing module is further configured to sequentially search for the plurality of log keywords in the ordered log sequence in time order.

[0066] The judging module is configured to judge whether the plurality of log keywords are all searched for.

[0067] The judging module is further configured to, in the case that the plurality of log keywords are all searched for, judge whether the order is consistent with the expected execution order of the predefined log procedure rule.

[0068] The determining module is further configured to, in the case that the order is consistent with the expected execution order of the predefined log procedure rule, determine that the actual execution path of the program under test is as expected.

[0069] Optionally, the processing module is further configured to count a triggering frequency of each business operation type within a preset period.

[0070] The determining module is further configured to determine a business operation type with a triggering frequency exceeding a preset frequency threshold as a high-frequency business operation type.

[0071] The processing module is further configured to cache a log template corresponding to the high-frequency business operation type to a local memory.

[0072] The processing module is further configured to read the log template from the local memory when the service operation type is a high-frequency service operation type, and use the log template to generate the predefined log format.

[0073] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory connected with the processor in communication;

[0074] The memory stores computer-executable instructions.

[0075] The processor executes the computer-executable instructions stored in the memory to implement the log processing method according to the first aspect and possible implementation manners of the first aspect.

[0076] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the log processing method according to the first aspect and possible implementation manners of the first aspect.

[0077] In a fifth aspect, the present application provides a program product, which comprises a computer program, and the computer program is executed by a processor to implement the log processing method.

[0078] The log processing method, device and electronic device provided by the present application are related to the field of big data technology. The method binds a test case with a log collection process, uses a "predefined log format + unique service identifier" to jointly construct a high-precision matching rule, and realizes automatic screening of associated logs in a distributed multi-server environment. Specifically, when responding to an execution request of a current test case, a predefined log format corresponding to the execution request is generated, and information of a plurality of servers related to the test case is acquired to determine a log collection source. After a test trigger, a globally unique service identifier is returned by a system under test, the identifier runs through the execution process of the current service operation in each service, the unique service identifier is embedded in a structure described by the predefined log format, a regular expression is constructed, and the regular expression is used to scan and match log files pulled from the plurality of servers. Only log records that meet the log format structure constraint and the unique service identifier consistency are retained, thereby screening a log set associated with the current test case, solving problems such as log dispersion, positioning difficulty and easy omission in a traditional manual manner in a distributed scenario, and improving the accuracy of log association and the collection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0079] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0080] Figure 1A flowchart illustrating a log processing method provided in this application. Figure 1 ;

[0081] Figure 2 A flowchart illustrating a log processing method provided in this application. Figure 2 ;

[0082] Figure 3 A flowchart illustrating a log processing method provided in this application. Figure 3 ;

[0083] Figure 4 A flowchart illustrating a log processing method provided in this application. Figure 4 ;

[0084] Figure 5 A schematic diagram of the structure of a log processing device provided in this application;

[0085] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application.

[0086] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0087] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0088] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, they do not violate public order and good morals, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0089] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0090] It should be noted that the log processing method, apparatus, and electronic device provided in this application can be used in the field of big data technology, or in any field other than big data. This application does not limit the application field of the log processing method, apparatus, and electronic device.

[0091] In software development and testing, especially in industries like finance and the internet where system stability and consistency are paramount, accurately tracing the execution path of business logic is crucial for ensuring system reliability. Typical scenarios include core banking systems or online payment platforms, where a single business operation often involves the collaborative processing of multiple distributed service components. Each component generates its own log file during execution. These logs record key information such as service calls, state changes, and business identifiers, forming the foundational data for reconstructing the complete execution flow. Developers and testers rely on log analysis to verify whether the actual execution path matches the intended design, thereby ensuring the correctness of system functionality and operational stability.

[0092] In existing technologies, log acquisition and analysis mainly rely on manual operation. That is, testers need to manually log in to multiple servers, filter log entries that may be related to the current test case from massive log data based on experience, and further manually compare the timestamps and business numbers in the logs of different servers to infer whether the actual execution logic of the program meets expectations.

[0093] While the aforementioned manual methods can meet basic analysis needs in simple scenarios, in large-scale distributed systems, as business logic becomes increasingly complex and the number of servers involved continues to increase, manual operations can no longer balance the efficiency and accuracy of log analysis, and cannot support the requirements of high-frequency, high-reliability automated testing.

[0094] Therefore, there is an urgent need for a method that can automatically associate logs from multiple servers, accurately identify the log records corresponding to test cases, and efficiently restore the program execution path, so as to achieve automation, structuring, and traceability of test verification in a distributed environment.

[0095] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0096] Figure 1 Flowchart of a log processing method provided by an embodiment of the present application Figure 1 As shown in Figure 1 , the log processing method provided by the embodiment includes:

[0097] S101, in response to an execution request of a current test case, a predefined log format is generated.

[0098] The predefined log format contains a fixed prefix and a field separation structure. The fixed prefix is usually composed of business identifiers, service module names or test scenario labels and other information, and is used to quickly distinguish different types of log sources; for example, in a payment system, the fixed prefix of a certain type of transaction log may be PAYMENT. The field separation structure defines the separation method (for example, using a colon) between the dynamic fields in the log, ensuring that the log content has a parsable structure, facilitating subsequent extraction of key information. The fixed prefix and the field separation structure provide convenience for machine processing of logs.

[0099] The log format is strongly bound to the test case, avoiding the technical defect of low efficiency caused by relying on fuzzy keywords or full-scan in traditional log analysis. The predefined log format not only reflects the expected output form of the business logic, but also provides a standardized position for the embedding of a unique business identifier. For example, when generating a regular expression for log matching, the fixed prefix serves as an anchor point, and the field separation structure guides how to locate the field where the transaction ID is located, thereby constructing a high-precision and low-false alarm matching rule.

[0100] The predefined log format improves the accuracy and efficiency of log screening and also lays a structured data foundation for the entire test verification process. This enables the automated test system to actively and intelligently work like "knowing what kind of log to look for", rather than passively searching blindly in massive texts, thereby significantly enhancing the reliability and maintainability of distributed system testing.

[0101] S102, a plurality of server information associated with the current test case is obtained.

[0102] In the single business operation, the business operation usually crosses multiple service nodes, and the execution trace of the business operation is scattered and recorded in the log files of multiple different servers. To realize the tracking of the complete execution link, the server information associated with the current test case is obtained in the test starting stage, and the server information is used to indicate the log collection source, and the server information includes the server IP address, the log path and the timestamp synchronization rule.

[0103] The core role of the server information is to determine the log collection source, that is, to indicate "which machines and which paths to read the logs from". Specifically, each piece of server information contains three key elements, for example: server IP address, log path and timestamp synchronization rule. Among them, the IP address is used to establish a network connection to access the target server; the log path is used to indicate the storage location of the log file to be collected, avoiding full disk scanning or path guessing, and improving the collection efficiency and reliability.

[0104] The timestamp synchronization rule contained in the server information is the key to solving the consistency problem of multiple server time sequences. Since there may be deviations in the system clocks of each server, directly using the original log timestamp for sorting may lead to misjudgment of the execution order. The timestamp synchronization rule is used to uniformly correct the log time, so as to obtain the time sequence log that truly reflects the business logic sequence.

[0105] By obtaining and utilizing the server information containing the IP address, log path and timestamp synchronization rule, the automatic positioning of the log collection source is realized, which lays a data foundation for subsequent cross-node log integration and time sequence correction.

[0106] S103, after triggering the test, obtaining the unique business identifier returned by the tested system.

[0107] Among them, after the test case is triggered for execution, the measured system returns a unique business identifier (such as a request serial number or a tracking ID). The unique business identifier has global uniqueness in the entire distributed call chain, and is usually created by the entry service of the measured system when receiving a test request, and is transmitted to all participating service modules as the business process advances. When each service module records the log, it will embed the unique business identifier in the log content as the "tag" of this operation in the node. The log entries originally scattered on multiple servers can be associated through the unique business identifier.

[0108] A possible implementation, in a cross-account transfer operation, the logs generated by the front-end gateway, the authentication service, the core account system and the notification service all contain the same serial number (such as Id=123456).

[0109] The unique service identifier is usually automatically obtained by a test framework through a standard interface (such as an API response header, a return body field or message queue metadata) of a system under test without manual intervention. The unique service identifier ensures that, in the log screening process, the system no longer depends on a fuzzy time window or keyword guess, but performs accurate matching based on a determined identity (the unique service identifier), thereby fundamentally improving the accuracy and reliability of log association.

[0110] In S104, a matching rule is constructed according to the unique service identifier and the predefined log format, and log records associated with the current test case are screened from log files of the plurality of servers.

[0111] The matching rule is constructed by the unique service identifier and the predefined log format, and is used to automatically screen log records associated with the current test case from massive logs of the plurality of servers. In one possible implementation, the predefined log format provides a structured template of the log, including a fixed prefix and a field separation structure, and the unique service identifier is a dynamic identity label of the current test. The unique service identifier is embedded into a corresponding field position defined by the log format, to generate a matching rule (regular expression) with explicit syntax constraints.

[0112] It is particularly pointed out that the matching rule not only requires the log to contain the specified unique service identifier, but also forces the overall structure to comply with the predefined log format, forming a "structure + identity" double-checking mechanism. Compared with the traditional method of relying only on keyword or simple string matching, the double-checking mechanism combined constraint mechanism can effectively exclude interference logs generated by other test cases, historical operations or irrelevant services, and improve the accuracy of the screening result.

[0113] In one possible implementation, the log files obtained from the plurality of servers are scanned in parallel by using the matching rule (regular expression), and a matching operation is performed line by line. A log line that meets the matching rule is determined to be triggered by the current test case, and the log line is reserved as a log record associated with the current test case.

[0114] The log processing method provided in this embodiment generates a predefined log format in response to an execution request of a current test case, obtains information of a plurality of servers associated with the current test case, obtains a unique service identifier returned by a system under test after triggering the test, constructs a matching rule according to the unique service identifier and the predefined log format, and screens log records associated with the current test case from log files of the plurality of servers, thereby solving the problems of log dispersion and manual positioning difficulty in a distributed environment, and improving the accuracy of log association and the collection efficiency.

[0115] Figure 2 Flowchart of a log processing method provided in an embodiment of the present applicationFigure 2 As shown in Figure 1 FIG. 1, the log processing method is described in detail based on the embodiment, including: Figure 3

[0116] S201, in response to an execution request of a current test case, determining a business operation type corresponding to the execution request.

[0117] When the test system receives an execution request of a current test case, it first determines the business operation type corresponding to the test case according to the business scenario simulated by the test case. For example, if the test case is used to verify the user login function, the business operation type is "user login"; if it is used to verify the fund transfer, the business operation type is "cross-bank transfer".

[0118] Different business operation types correspond to different service call chains and log output structures. For example, "payment success" and "payment failure" belong to the payment business, but their log content, state field and expected process are significantly different, and need to be configured with matching rules respectively. The business operation type is used as a key identifier to find the log processing specification matched therewith from the pre-configured rule library.

[0119] By determining the business operation type, the matching log processing strategy can be loaded for the current test case, avoiding the use of general or incorrect templates leading to log missing or misjudgment.

[0120] S202, based on the business operation type, obtaining a corresponding log template from a pre-configured rule library, the log template defining a fixed prefix and a field separation structure.

[0121] S203, generating a predefined log format according to the log template.

[0122] The rule library is a centrally managed configuration set, which pre-defines the corresponding log template for each business operation type. The log template specifies the format specification that should be followed by the business when outputting logs in each service component. Specifically, the log template defines two elements: one is the fixed prefix, i.e. the string at the beginning of the log, which usually contains the test identification, business category and service module information; the other is the field separation structure, i.e. the separation method used between the information fields in the log, for example, using colon, vertical line or space for separation, and specifying the arrangement order of the fields. Through this template, different services can generate log texts with consistent structure and clear semantics when recording the same business operation.

[0123] ​After determining the business operation type, the test system accesses a preset rule library. The rule library is organized in the form of key-value pairs, where the "key" is the business operation type name, and the "value" is the corresponding log template. The log template is pre-recorded by the test personnel or operation and maintenance personnel during the system deployment stage according to the actual business log specification.

[0124] After reading the log template, the fixed prefix and separator in the log template are extracted to form a structured format description. The format description specifies that the target log must start with the "fixed prefix", and the fields in the log (such as service name, transaction ID, status, etc.) must be strictly arranged according to the "separator".

[0125] S204, obtain a plurality of server information associated with the current test case, the server information being used to indicate a log collection source, and the server information including a server IP address, a log path, and a timestamp synchronization rule.

[0126] Step S204 is similar to step S102 described above, and will not be described here again.

[0127] S205, after triggering the test, obtaining a unique business identifier returned by the tested system.

[0128] Step S205 is similar to step S103 described above, and will not be described here again.

[0129] S206, based on the server IP address and the log path, obtaining log files from a plurality of servers.

[0130] Among them, according to the obtained plurality of server information, log file pulling operations are performed on each target server in the plurality of servers in turn. In one possible implementation, for each server, a remote connection is established using its IP address. The connection can be achieved through a standard protocol, such as secure login through the SSH protocol, or file transfer through the SFTP / SCP protocol.

[0131] After establishing the connection, the specified log file is read according to the log path corresponding to the server. The log path is pre-configured and accurately points to the location where the tested service actually writes the log. By directly accessing the log file under the log path, invalid operations caused by scanning the entire directory can be avoided, and the collection efficiency is improved.

[0132] It is particularly pointed out that the above-mentioned obtaining of log files from servers can be performed in parallel. Specifically, connection requests are initiated to a plurality of IP addresses simultaneously, and log files are obtained according to the corresponding log paths. All log files are collected into a cache area for subsequent unified processing.

[0133] S207, generating a matching pattern based on the fixed prefix and field separation structure in the predefined log format.

[0134] S208, generating a regular expression according to the matching pattern and the unique service identifier.

[0135] Wherein, the predefined log format is parsed, the fixed prefix and the field separation structure are extracted. The fixed prefix is a string unchanged at the beginning of the log line, which is used to quickly identify the test scenario and service type to which the log belongs; the field separation structure indicates the separation method between the information fields in the log, and implies the arrangement order of the fields.

[0136] According to the fixed prefix and the field separation structure, a matching pattern is generated, which defines the basic format constraints that the target log must satisfy. For example, if the fixed prefix is "AUTO-TEST:TRANSFER", the field separator is ":", and it is known that the transaction ID appears in the fourth field position, the matching pattern can be expressed as: "The log line must start with 'AUTO-TEST:TRANSFER', followed by a number of fields separated by colons, and the fourth field is in the form of 'txnId=xxx'".

[0137] The unique service identifier (such as "123456") is embedded in the above matching pattern to generate a specific regular expression. Specifically, in the transaction ID field position indicated by the matching pattern, the placeholder is replaced by the unique service identifier value, and the special characters are escaped to ensure the legality of the regular syntax. The final generated regular expression not only requires the log to contain the specified unique service identifier, but also requires its overall structure to conform to the predefined log format, thus forming a "structure + identity" double verification mechanism.

[0138] Through the above processing, the abstract business configuration is converted into an executable text matching rule, which provides technical support for subsequent filtering of log records associated with the current test case from massive logs.

[0139] S209, compiling the regular expression into a regular matching object.

[0140] S210, reading the log file of each server line by line, and calling the regular matching object to perform matching operation on the current line log text.

[0141] Wherein, after generating the regular expression, the programming language is called to compile the regular expression into a regular matching object. The matching object is an internal optimized representation of the regular expression, whose structure is preprocessed and state machine converted, which can be reused in subsequent multiple matching operations, avoiding re-parsing of the regular syntax every time, thereby significantly improving the processing performance.

[0142] The log files obtained from each server are read line by line. Specifically, for each log file, each line of text content is read in a streaming manner in sequence, ensuring that even in the face of large files, the reading will not be interrupted due to insufficient memory. For each line of log text read, the aforementioned compiled regular matching object is immediately called to perform a matching operation. This operation determines whether the current log line completely meets the structure and content constraints defined by the regular expression, that is, whether it simultaneously meets the fixed prefix, field separation format, and contains the specified unique business identifier.

[0143] S211, if the matching operation returns success, the current line of log text is retained.

[0144] S212, if the matching operation returns failure, the current line of log text is skipped.

[0145] S213, the matching successful log is taken as a log record associated with the current test case.

[0146] If the matching operation returns success (i.e., the log line completely matches the regular expression), the log line is retained and added to the log set; if the matching fails, the line is directly discarded without further processing. All matching successful log lines are stored as log records associated with the current test case.

[0147] Through this streaming processing mechanism, while ensuring filtering accuracy, memory occupation and invalid calculation are maximally reduced.

[0148] The log processing method provided in this embodiment determines the corresponding business operation type in response to the execution request of the test case, loads the corresponding log template from the pre-set rule library, generates a structured pre-defined log format; at the same time, obtains multiple server information (including IP address, log path and timestamp synchronization rule) associated with the test case, and obtains the unique business identifier returned by the system under test after the test is triggered, constructs a regular expression containing a fixed prefix, field separation structure and unique business identifier, compiles the regular expression into a matching object, and performs line-by-line matching processing on the log files pulled from each server, only retains the matching successful log line as a log record associated with the current test case, realizes automatic log collection, avoids manual screening, and improves test efficiency and verification reliability.

[0149] Figure 3 Flowchart of a log processing method provided in an embodiment of the present application Figure 1 . As shown in Figure 4 , on the basis of the Figure 4 embodiment, the log processing method is described in detail, including:

[0150] S301, extract the original timestamp of the filtered log record and the associated server identifier.

[0151] Among them, for each filtered log record, the text content is parsed, and the original timestamp is extracted. The original timestamp is the time information written by the corresponding server local system when the log is generated, usually located in a fixed position of the log line. According to the position and format agreement of the time field in the pre-defined log format, the time value is accurately identified and extracted as the original basis for subsequent time correction.

[0152] At the same time, the server identifier associated with each log record is extracted. The server identifier is used to uniquely indicate the log source. One possible implementation, when pulling logs from a server, the IP address, hostname or service instance ID of the server is attached to each log record as metadata.

[0153] The extracted original timestamp and server identifier are saved in pairs as key inputs for subsequent time correction. Because there may be system clock drifts between different servers, directly using the original timestamp for sorting may lead to misjudgment of the execution order. Therefore, combined with the server identifier, the corresponding clock calibration parameters are found, so that the timestamp can be uniformly corrected, thereby constructing a global ordered log sequence that truly reflects the business logic sequence.

[0154] S302, according to the server identifier, obtain the timestamp synchronization rule of the server, and correct the original timestamp based on the timestamp synchronization rule to obtain the corrected timestamp.

[0155] S303, sort the log records according to the corrected timestamp to obtain an ordered log sequence.

[0156] Among them, with the server identifier (such as IP address, hostname or service instance ID) as the index, the timestamp synchronization rule corresponding to the server is queried from the pre-configured server information set. According to the timestamp synchronization rule, the original timestamp in the log is mathematically transformed or format converted, so as to calculate the corrected timestamp. For example, if a server is 150 milliseconds faster, its synchronization rule is "-150ms", then the system will uniformly subtract 150 milliseconds from the original timestamp of all logs on this server, so that it is aligned with the time reference of other servers.

[0157] Through correction, the distorted log timing due to the different clock synchronization of multiple machines is restored, ensuring that the subsequent sorting result truly reflects the actual execution sequence of business operations.

[0158] After completing the time correction of all logs, all associated log records are sorted in ascending order according to the corrected timestamp to generate a globally consistent ordered log sequence. The ordered log sequence eliminates the interference caused by clock drift.

[0159] S304, a predefined log flow rule associated with the current test case is obtained, and the predefined log flow rule includes a plurality of log keywords arranged in an expected execution order.

[0160] The predefined log flow rule associated with the current test case is obtained. The predefined log flow rule is a structured verification basis pre-configured by a tester according to business logic in a test design stage, and is usually uniquely bound to the test case.

[0161] The core content of the predefined log flow rule is a plurality of log keywords arranged in an expected execution order. The log keywords represent the symbolic log content of the key nodes in the business flow. For example, in the "user cross-line transfer" scenario, the expected key steps can include "receiving transfer request", "verifying account balance", "calling clearing service", "updating transfer-out account", "updating transfer-in account", and "sending transaction success notification". Each step corresponds to one or more log keywords with recognition degree and is strictly arranged in the order of business logic to form a linear and verifiable flow chain.

[0162] By reading the identification information of the current test case, the corresponding predefined log flow rule is retrieved from the rule library to determine the expected execution order to be followed in the current verification. The predefined log flow rule not only defines "which keywords must appear", but also emphasizes "they must appear in which order".

[0163] S305, a plurality of log keywords in the ordered log sequence are sequentially searched in time order.

[0164] S306, it is judged whether the plurality of log keywords are all found, and if so, step S307 is performed.

[0165] S307, it is judged whether the order is consistent with the expected execution order of the predefined log flow rule, and if so, step S308 is performed.

[0166] S308, it is determined that the actual execution path of the program under test conforms to the expectation.

[0167] The plurality of log keywords specified in the predefined log flow rule are sequentially searched in time order. Specifically, starting from the first log of the ordered log sequence, the content of each log is scanned and matched with the first keyword in the flow rule. Once a match is found, the position is locked, and the search for the second keyword in the flow rule continues from the next log. In this way, all the to-be-searched keywords are traversed or the ordered log sequence ends.

[0168] After all the searching is completed, it is determined whether the multiple log keywords are all found. That is, it is determined whether each keyword defined in the flow rule is successfully matched at least once in the ordered log sequence in order. If all the keywords are found and the order of appearance is completely consistent with the predefined rule, it is considered that the execution path of the program under test conforms to the expectation; if any keyword is not found or exists but is in disorder, it is determined that the path deviates.

[0169] The log processing method provided in the embodiment corrects the time and verifies the flow on the basis of completing the screening of the log records associated with the current test case. Specifically, the original time stamp in the screened log record and the server identifier to which the log record belongs are extracted; the original time stamp is uniformly corrected according to the time stamp synchronization rule of each server, so as to eliminate the time sequence deviation caused by the unsynchronized system clock; the log records are globally sorted according to the corrected time stamp, so as to obtain an ordered log sequence reflecting the real execution sequence; the predefined log flow rule associated with the current test case is obtained, the rule defines multiple log keywords that should appear in turn in the expected execution process of the business operation in a structured form; the log keywords are searched in the ordered log sequence in time sequence, and it is determined whether all the log keywords exist and the order of appearance is consistent with the predefined rule; if the condition is met, it is determined that the actual execution path of the program under test conforms to the expected design, the automatic verification of the actual execution path of the program is realized, the flow misjudgment problem caused by the unsynchronized clock is avoided, and the accuracy and reliability of the test result are improved.

[0170] Figure 1 Flow diagram of the log processing method provided in the embodiment Figure 5 . As shown in Figure 5 , on the basis of the Figure 6 embodiment, the log processing method is described in detail, including:

[0171] S401, the triggering frequency of each business operation type in a preset period is counted.

[0172] S402, the business operation type with a triggering frequency exceeding a preset frequency threshold is determined as a high-frequency business operation type.

[0173] In which, the triggering frequency of each business operation type in a preset period is counted continuously. The preset period may be, for example, the last 24 hours or a time window configured by an operation and maintenance personnel according to the actual load condition. In one possible implementation, when a test case is executed, the corresponding business operation type is identified, and the call frequency is added to the corresponding counter once.

[0174] The trigger frequency of each business operation type is compared with a preset frequency threshold. The threshold can be configured according to system resources, test scale or performance target, for example, set to "more than 100 times per hour". The business operation type with a trigger frequency exceeding the threshold is marked as a high-frequency business operation type.

[0175] S403, cache the log template corresponding to the high-frequency business operation type to the local memory.

[0176] S404, when the business operation type is a high-frequency business operation type, read the log template from the local memory for generating the predefined log format.

[0177] Wherein, after identifying the high-frequency business operation type, the log template corresponding to the high-frequency business operation type is loaded once from the preset rule library and cached to the special cache area in the local memory.

[0178] Whenever a new test case is executed, it is judged whether the business operation type corresponding to the test case execution request belongs to the marked high-frequency business operation type. If yes, the cached log template is directly read from the local memory without accessing the external rule library, and the predefined log format is generated based on the cached log template read from the local memory.

[0179] Non-high-frequency business operation types still load log templates from the rule library as needed to ensure functional integrity, but do not occupy valuable memory resources. Through this "hot business operation type caching and cold business operation type loading as needed" strategy, the processing efficiency of high-frequency test scenarios is improved.

[0180] The log processing method provided in the embodiment identifies high-frequency business operation types by counting the trigger frequency of each business operation type, and caches the log template corresponding to the high-frequency business operation type to the local memory. In subsequent processing, if it is detected that the current test case belongs to the high-frequency type, the log template is directly read from the memory to generate the predefined log format, reducing repeated access to the rule library and improving the log format generation efficiency.

[0181] Figure 6 The structure diagram of a log processing device provided in the present application is shown in FIG. 1. As shown in FIG. 1, the present application provides a log processing device 500, which comprises: ​

[0182] The processing module 501 is configured to generate a predefined log format in response to an execution request of a current test case, and the predefined log format comprises a fixed prefix and a field separation structure.

[0183] ​The acquisition module 502 is configured to acquire a plurality of server information associated with the current test case, the server information being used to indicate a log collection source, and the server information including a server IP address, a log path, and a timestamp synchronization rule.

[0184] The acquisition module 502 is further configured to acquire a unique service identifier returned by the tested system after triggering the test.

[0185] The processing module 501 is further configured to construct a matching rule according to the unique service identifier and the predefined log format, and filter log records associated with the current test case from log files of the plurality of servers.

[0186] Optionally, the apparatus further includes a determination module 503.

[0187] The determination module 503 is configured to determine a service operation type corresponding to an execution request of the current test case in response to the execution request.

[0188] The acquisition module 501 is further configured to acquire a corresponding log template from a preset rule library based on the service operation type, the log template defining a fixed prefix and a field separation structure.

[0189] The processing module 502 is further configured to generate the predefined log format according to the log template.

[0190] Optionally, the acquisition module 501 is further configured to acquire log files from the plurality of servers based on the server IP address and the log path.

[0191] The processing module 502 is further configured to generate a matching pattern based on the fixed prefix and the field separation structure in the predefined log format.

[0192] The processing module 502 is further configured to generate a regular expression according to the matching pattern and the unique service identifier.

[0193] The processing module 502 is further configured to perform matching processing on the log files based on the regular expression.

[0194] The processing module 502 is further configured to take log records matched successfully as log records associated with the current test case.

[0195] Optionally, the processing module 502 is further configured to compile the regular expression into a regular matching object.

[0196] The processing module 502 is further configured to read the log files of each server line by line, and call the regular matching object to perform a matching operation on current line log text.

[0197] The processing module 502 is further configured to retain the current line log text if the matching operation returns success.

[0198] The processing module 502 is also used to skip the current line of log text if the matching operation returns a failure.

[0199] Optionally, the processing module 502 is also used to extract the original timestamps of the filtered log records and the associated server identifiers;

[0200] The acquisition module 501 is also used to obtain the server's timestamp synchronization rules based on the server identifier;

[0201] The processing module 502 is also used to correct the original timestamp based on the timestamp synchronization rules to obtain the corrected timestamp;

[0202] The processing module 502 is also used to sort the log records according to the corrected timestamps to obtain an ordered log sequence.

[0203] Optionally, the device may also include: a judgment module 504;

[0204] The acquisition module 501 is also used to acquire predefined log flow rules associated with the current test case. The predefined log flow rules include multiple log keywords arranged in the expected execution order.

[0205] Processing module 502 is also used to sequentially search for multiple log keywords in an ordered log sequence according to time order;

[0206] The judgment module 504 is used to determine whether multiple log keywords have been found.

[0207] The judgment module 504 is also used to determine whether the order of judgment is consistent with the expected execution order of the predefined log flow rules when multiple log keywords have been found.

[0208] The determination module 503 is also used to determine whether the actual execution path of the program under test conforms to expectations, provided that the sequence is consistent with the expected execution order of the predefined log flow rules.

[0209] Optionally, the processing module 502 is also used to count the trigger frequency of each business operation type within a preset time period;

[0210] The determination module 503 is also used to determine the business operation type whose trigger frequency exceeds the preset frequency threshold as a high-frequency business operation type;

[0211] The processing module 502 is also used to cache the log templates corresponding to high-frequency business operation types to local memory;

[0212] The processing module 502 is also used to read the log template from local memory when the business operation type is a high-frequency business operation type, and to generate a predefined log format.

[0213] The log processing apparatus provided by the embodiments of the present application has similar implementation principles and technical effects to the implementation manners of the various parts of the aforementioned log processing method, and thus will not be described again here.

[0214] ​ A structural schematic diagram of an electronic device is provided in the present application. As shown in the figure, the electronic device 600 includes a receiver 601, a transmitter 602, a processor 603, and a memory 604. ​ The receiver 601 is configured to receive instructions and data.

[0215] The transmitter 602 is configured to transmit instructions and data.

[0216] The memory 604 is configured to store computer-executed instructions.

[0217] The processor 603 is configured to execute the computer-executed instructions stored in the memory 604, so as to implement the various steps performed by the log processing method in the aforementioned embodiments. For details, please refer to the related descriptions in the aforementioned embodiments of the log processing method.

[0218] Optionally, the memory 604 can be independent or integrated with the processor 603.

[0219] When the memory 604 is independently arranged, the electronic device further includes a bus configured to connect the memory 604 and the processor 603.

[0220] The implementation principles and technical effects of the electronic device provided by the embodiments can be referred to the aforementioned embodiments, and thus will not be described again here.

[0221] The embodiments of the present application further provide a computer-readable storage medium, which stores computer-executed instructions. When the processor executes the computer-executed instructions, the method of any one of the aforementioned embodiments is implemented.

[0222] The embodiments of the present application further provide a computer program product, which includes a computer program. When the processor executes the computer program, the method of any one of the aforementioned embodiments is implemented.

[0223] It should be noted that, for the aforementioned method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0224] It should be noted that, for the aforementioned method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0225] It should be further understood that, in the flow charts, each step is shown in a sequential order, but the steps are not necessarily performed in the order shown by the arrows. Unless otherwise specifically noted, the steps can be performed in any order, and the steps can be performed in other orders. Also, at least some of the steps in the flow charts can include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, and which can be performed in different orders, and which can be performed in rotation or alternation with at least some of the steps or sub-steps or stages of other steps.

[0226] It should be understood that the above-mentioned apparatus embodiments are only illustrative, and the apparatus of the present application can also be implemented in other manners. For example, the division of the units / modules in the above-mentioned embodiments is only a logical function division, and actual implementation can be in another division manner. For example, multiple units / modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0227] In addition, unless specifically stated, each functional unit / module in each embodiment of the present application can be integrated in one unit / module, or can be physically present separately, or two or more units / modules can be integrated together. The above-mentioned integrated unit / module can be realized in the form of hardware or in the form of a software program module.

[0228] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. Each technical feature of the above-mentioned embodiments can be combined arbitrarily, and in order to make the description brief, not all possible combinations of the technical features in the above-mentioned embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0229] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the general inventive concepts described herein. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application indicated by the following claims.

[0230] It should be understood that the present application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be effected therein by those skilled in the art without departing from the scope of the application. The scope of the application should be limited only by the appended claims.

Claims

1. A log processing method, characterized in that, The method includes: In response to the execution request of the current test case, a predefined log format is generated, which includes a fixed prefix and a field separator structure; Obtain information about multiple servers associated with the current test case. The server information is used to indicate the log collection source and includes the server IP address, log path, and timestamp synchronization rules. After triggering the test, obtain the unique business identifier returned by the system under test; Based on the unique business identifier and the predefined log format, a matching rule is constructed to filter out log records associated with the current test case from the log files of the multiple servers.

2. The method according to claim 1, characterized in that, The step of generating a predefined log format in response to the execution request of the current test case includes: In response to the execution request of the current test case, determine the business operation type corresponding to the execution request; Based on the business operation type, a corresponding log template is obtained from a pre-set rule base. The log template defines the fixed prefix and the field separator structure. The predefined log format is generated based on the log template.

3. The method according to claim 1, characterized in that, The step of constructing matching rules based on the unique business identifier and the predefined log format, and filtering log records associated with the current test case from the log files of the multiple servers, includes: Log files are obtained from the multiple servers based on the server IP address and log path. A matching pattern is generated based on the fixed prefix and field separator structure in the predefined log format. Generate a regular expression based on the matching pattern and the unique business identifier; The log file is matched based on the regular expression. Logs that match successfully will be recorded as log entries associated with the current test case.

4. The method according to claim 3, characterized in that, The matching process for the log file based on the regular expression includes: Compile the regular expression into a regular expression matching object; Read the log file of each server line by line, and call the regular expression matching object to perform a matching operation on the current line of log text; If the matching operation returns successfully, the current line of log text is retained; If the matching operation fails, the current line of log text is skipped.

5. The method according to claim 1, characterized in that, The method further includes: Extract the original timestamps and associated server identifiers of the filtered log records; Based on the server identifier, the timestamp synchronization rule of the server is obtained, and the original timestamp is corrected based on the timestamp synchronization rule to obtain the corrected timestamp; The log records are sorted according to the corrected timestamps to obtain an ordered log sequence.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the predefined log flow rules associated with the current test case, the predefined log flow rules including multiple log keywords arranged in the expected execution order; Search for the multiple log keywords in the ordered log sequence in chronological order; Determine whether all of the log keywords have been found; If so, determine whether the order is consistent with the expected execution order of the predefined log flow rules; If the sequence is consistent with the expected execution order of the predefined log flow rules, it is determined that the actual execution path of the program under test conforms to expectations.

7. The method according to claim 2, characterized in that, Before generating the predefined log format based on the log template, the method further includes: Statistical analysis of the trigger frequency of each business operation type within a preset time period; The service operation type whose trigger frequency exceeds a preset frequency threshold is identified as a high-frequency service operation type; Cache the log templates corresponding to the high-frequency business operation types in local memory; When the business operation type is a high-frequency business operation type, the log template is read from the local memory and used to generate the predefined log format.

8. A log processing device, characterized in that, The device includes: The processing module is used to respond to the execution request of the current test case and generate a predefined log format, which includes a fixed prefix and a field separator structure. The acquisition module is used to acquire information about multiple servers associated with the current test case. The server information is used to indicate the log collection source and includes the server IP address, log path, and timestamp synchronization rules. The acquisition module is also used to acquire the unique business identifier returned by the system under test after the test is triggered; The processing module is further configured to construct matching rules based on the unique business identifier and the predefined log format, and to filter out log records associated with the current test case from the log files of the multiple servers.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images