Log Processing Method, Device, Equipment, Medium and Program Product

Abstract

The present disclosure provides a log processing method, which can be applied to the field of information security or the financial field. The method includes: obtaining a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation; based on the content of the first log, obtaining N second logs, where each second log includes log information generated by testing a first business scenario, and the first business scenario is any one of M business scenarios; obtaining the similarity between each second log and a log template; for each second log, when the similarity meets a first preset condition, confirming that the second log is of the same category as the log template. The present disclosure also provides a log processing device, equipment, storage medium, and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of information security or the field of finance, and more particularly to a log processing method, apparatus, device, medium, and program product. Background Art

[0002] Out of considerations for aspects such as the correctness, integrity, security, and stability of the product, test verification can be performed before the product is put into production to determine whether the expected requirements are met. Taking the transaction processing system of a bank as an example, automated testing can be performed by executing multiple transactions through automated scripts. For some complex transactions that cannot be implemented through automated scripts, manual testing can be performed. During the testing process, automated testing and manual testing can be carried out simultaneously, and test logs are output.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: Since the logs output by automated testing and manual testing are mixed together, but the number of transactions in automated testing and manual testing may not be the same, the amount of log content output is also different. For the party with fewer test transactions, when a certain transaction goes wrong, it is difficult to accurately locate the log content of that transaction, resulting in the inability to quickly troubleshoot problems and low test efficiency. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a log processing method, apparatus, device, medium, and program product for improving test efficiency.

[0005] In one aspect of the embodiments of the present disclosure, a log processing method is provided, including: obtaining a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation; obtaining N second logs based on the content of the first log, where each second log includes log information generated by testing a first business scenario, and the first business scenario is any one of M business scenarios, and N and M are integers greater than or equal to 1; obtaining the similarity between each second log and a log template, where the log template is obtained based on the log generated by the automated test operation or the manual test operation; for each second log, when the similarity meets a first preset condition, confirming that the second log is of the same category as the log template, where the category of the log template includes an automated category or a manual category.

[0006] According to an embodiment of the present disclosure, the obtaining N second logs based on the content of the first log includes: copying the content of the first log to obtain a third log cluster, where the third log cluster includes S third logs, and S is an integer greater than or equal to 1; obtaining N second logs according to the content of the third log cluster.

[0007] According to an embodiment of the present disclosure, obtaining N second logs according to the content of the third log cluster includes obtaining each second log, specifically including: obtaining a first start identifier in the third log, where the first start identifier is any start identifier in the third log; obtaining a first thread symbol according to the first start identifier, where the first start identifier and the first thread symbol are in the first log information in the third log; starting from the first log information, sequentially obtaining at least one log information including the first thread symbol; writing the at least one log information into the second log, where the last log information in the at least one log information includes a first end identifier.

[0008] According to an embodiment of the present disclosure, it further includes: for each second log, if it is of the same category as the log template, write it into the fourth log; if it is not of the same category as the log template, write it into the fifth log.

[0009] According to an embodiment of the present disclosure, it further includes: matching the fourth log with the first log; deleting the content identical to the fourth log from the first log to obtain a sixth log.

[0010] According to an embodiment of the present disclosure, overwriting the content of the first log to obtain the third log cluster includes: monitoring the data volume of the first log; when the growth rate of the data volume meets a second preset condition, overwriting the content of the first log, where the second preset condition includes that the growth rate of the data volume is greater than or equal to a first preset threshold.

[0011] According to an embodiment of the present disclosure, overwriting the content of the first log to obtain the third log cluster further includes: based on the timestamp when the growth rate of the data volume meets the second preset condition, writing the log information generated after the timestamp into the i-th third log, where i is an integer greater than or equal to 1 and less than or equal to S; when the data volume of the i-th third log is greater than or equal to a second preset threshold, writing the log information generated after the timestamp and not overwritten into the (i + 1)-th third log.

[0012] According to an embodiment of the present disclosure, it further includes: monitoring the number of third logs in the third log cluster; when the growth rate of the number meets a third preset condition, stop overwriting the content of the first log, where the second preset condition includes that the growth rate of the number is less than a second preset threshold.

[0013] According to an embodiment of the present disclosure, obtaining the similarity between each of the second logs and the log template includes: matching the buried point information in each of the second logs with the preset buried point information of the log template; and obtaining the similarity based on the number of successful matches.

[0014] According to an embodiment of the present disclosure, before obtaining the similarity between each of the second logs and the log template, obtaining the log template is further included, which specifically includes: separately performing the automated test operation or separately performing the manual test operation to generate a seventh log; and writing at least one of the preset buried point information into the seventh log to obtain the log template.

[0015] According to an embodiment of the present disclosure, it includes: storing at least one first address information and the category of each first address information, where the first address information is used to mark the client address where the test operation is executed; before obtaining the similarity between each of the second logs and the log template, the method further includes: for each of the second logs, in the case where there is a second address information, if the second address information matches any one of the first address information, confirming that the second log is of the same category as the first address information it matches.

[0016] Another aspect of the embodiments of the present disclosure provides a log processing device, including: a first obtaining module, configured to obtain a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation; a second obtaining module, configured to obtain N second logs based on the content of the first log, where each of the second logs includes log information generated by testing a first service scenario, and the first service scenario is any one of M service scenarios, and N and M are integers greater than or equal to 1; a log cleaning module, configured to obtain the similarity between each of the second logs and the log template, where the log template is obtained based on the log generated by the automated test operation or the manual test operation; and a log classification module, configured to, for each of the second logs, in the case where the similarity meets a first preset condition, confirm that the second log is of the same category as the log template, where the category of the log template includes an automated category or a manual category.

[0017] Another aspect of the embodiments of the present disclosure provides an electronic device, including: one or more processors; a storage device, configured to store one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method as described above.

[0018] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is caused to execute the method as described above.

[0019] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program which, when executed by a processor, implements the method as described above.

[0020] One or more of the above embodiments have the following beneficial effects: Based on the dimension of the business scenario, N second logs are obtained from the first log, and each second log is classified. By using the similarity between each second log and the log template as the classification basis, when the similarity meets the first preset condition, it is confirmed that the category of the second log is the same as that of the log template. The content in the first log can be accurately classified. Whether there are problems in transactions during automated testing or manual testing, the log content of the corresponding transaction can be accurately located through the logs of the corresponding category, improving the testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above content and other objects, features and advantages of the present disclosure will become clearer. In the drawings:

[0022] Figure 1 Schematically shows an application scenario diagram of the log processing method according to an embodiment of the present disclosure;

[0023] Figure 2 Schematically shows a flowchart of the log processing method according to an embodiment of the present disclosure;

[0024] Figure 3 Schematically shows a flowchart of obtaining the second log according to an embodiment of the present disclosure;

[0025] Figure 4 Schematically shows a flowchart of obtaining the second log according to another embodiment of the present disclosure;

[0026] Figure 5 Schematically shows a flowchart of obtaining the sixth log according to an embodiment of the present disclosure;

[0027] Figure 6 Schematically shows a flowchart of the log processing method according to another embodiment of the present disclosure;

[0028] Figure 7 Schematically shows a flowchart of obtaining the third log cluster according to an embodiment of the present disclosure;

[0029] Figure 8 Schematically shows a flowchart of obtaining the third log cluster according to another embodiment of the present disclosure;

[0030] Figure 9 Schematically shows a flowchart of stopping overwriting the first log according to an embodiment of the present disclosure;

[0031] Figure 10 Schematically shows a flowchart of obtaining similarity according to an embodiment of the present disclosure;

[0032] Figure 11 Schematically shows a flowchart of obtaining a log template according to an embodiment of the present disclosure;

[0033] Figure 12 Schematically shows a flowchart of a log processing method according to another embodiment of the present disclosure;

[0034] Figure 13 Schematically shows a structural block diagram of a log processing apparatus according to an embodiment of the present disclosure;

[0035] Figure 14 Schematically shows a block diagram of an electronic device suitable for implementing the log processing method according to an embodiment of the present disclosure. Detailed implementation manners

[0036] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0037] In the daily enterprise testing work, the proportion of automated testing is increasing. While machines simulate humans for automated testing, manual testing is also being carried out, and a large number of transaction logs will be generated in the background. The logs of automated testing and manual testing are mixed together.

[0038] Taking the case where the number of transactions in automated testing is more than that in manual testing as an example, when an online business transaction during manual testing goes wrong, it is very difficult for technicians to quickly and accurately locate the log content. Because the volume of automated testing logs is large and the refresh speed is fast, it is impossible to quickly locate the manual online transaction logs in a large number of logs. In addition, the overall storage capacity of the test logs is limited, and the superposition of a large number of automated testing logs in a short period of time is also likely to trigger capacity overrun, and the previous manual online transaction logs may be completely cleared.

[0039] Especially in some account-related transactions, many business transactions also have characteristics such as a long pre-data preparation cycle and irreversibility of transactions. It is also very difficult to reproduce the same problem by manually operating the same business transaction again. Therefore, the problem cannot be analyzed and solved, which has a greater impact.

[0040] Embodiments of the present disclosure provide a log processing method, which includes: obtaining a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation. Based on the content of the first log, obtaining N second logs, where each second log includes log information generated by testing a first business scenario, and the first business scenario is any one of M business scenarios, and N and M are integers greater than or equal to 1 respectively. Obtaining the similarity between each second log and a log template, where the log template includes logs generated by an automated test operation or a manual test operation. For each second log, when the similarity meets a first preset condition, it is confirmed that the second log has the same category as the log template, where the category of the log template includes an automated category or a manual category.

[0041] According to the embodiments of the present disclosure, based on the dimension of the business scenario, N second logs are obtained from the first log, and each second log is classified. Using the similarity between each second log and the log template as the classification basis, when the similarity meets the first preset condition, it is confirmed that the second log has the same category as the log template. The content in the first log can be accurately classified. Whether there is a problem with a transaction in an automated test or a manual test, the log content of this transaction can be accurately located through the logs of the corresponding category, improving the test efficiency.

[0042] It should be noted that a log processing method, device, equipment, medium, and program product involved in the present disclosure can be used in aspects related to log processing for product testing in the financial field, and can also be used in test scenarios in the financial field or other fields. The present disclosure does not limit the application field.

[0043] Figure 1 An application scenario diagram of the log processing method according to the embodiments of the present disclosure is schematically shown.

[0044] As Figure 1 shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0045] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0046] The terminal devices 101, 102, and 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and so on.

[0047] The server 105 can be a server providing various services, such as a background management server (only for example) that supports the websites browsed by users using the terminal devices 101, 102, and 103. The background management server can analyze and process data such as user requests received, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0048] According to an embodiment of the present disclosure, the test operation can be implemented through a B / S architecture (browser / server architecture). Specifically, middleware can be set on the basis of the B / S architecture to implement the output and storage of test logs. The middleware can be, for example, a WAS (Websphere Application Server) architecture.

[0049] It should be noted that the log processing method provided by the embodiment of the present disclosure can generally be executed by the server 105. Correspondingly, the log processing device provided by the embodiment of the present disclosure can generally be set in the server 105. The log processing method provided by the embodiment of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the log processing device provided by the embodiment of the present disclosure can also be set in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0050] It should be understood that Figure 1 the numbers of the terminal devices, the network, and the server in

[0051] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers. Figure 1 The following will be based on Figures 2 to 12 the described scenario, and will describe in detail the log processing method of the embodiment of the present disclosure through

[0052] Figure 2 FIG. schematically shows a flowchart of the log processing method according to an embodiment of the present disclosure.

[0053] As Figure 2 shown, the log processing method of this embodiment includes operations S210 to S240.

[0054] In operation S210, a first log generated in response to a test operation is obtained, where the test operation includes an automated test operation and a manual test operation.

[0055] The test operation can be a software test, which is a process of using manual or automated means to run or measure a software system, aiming to check whether it meets the specified requirements or to find out the difference between the expected result and the actual result. The manual test operation refers to the execution of part or all of the test process by humans. The automated test operation means that after starting the automated script, most of the processes can be executed and almost no human participation is required.

[0056] The first log can include the application log of the output of the WAS architecture application, which can be named SystemOut.log (for example only). Each log message is accompanied by a hexadecimal value formed by 8 characters for marking the thread, and the log message records the application activity information. Among them, each log message can be each segment of the log content in the first log.

[0057] In operation S220, based on the content of the first log, N second logs are obtained, where each second log includes the log information generated by testing a first business scenario, and the first business scenario is any one of the M business scenarios. N and M are integers greater than or equal to 1 respectively.

[0058] The first log can include the application activity information of each process of the object under test in processing each transaction during the test operation. The content of the first log can be split to obtain N second logs. The M business scenarios can include M test transactions executed during the test operation. Each business scenario corresponds to each transaction and has corresponding log information. The second log can include the log information of a transaction.

[0059] In operation S230, the similarity between each second log and the log template is obtained, where the log template is obtained based on the log generated by the automated test operation or the manual test operation.

[0060] The information such as the user name, IP address, business name, automation identifier, paragraph format, etc. included in the log information output by the transactions in the automated test operation is different from the log information output by the transactions in the manual test operation. Therefore, in the case where the log template is obtained based on the log generated by either the automated test operation or the manual test operation, the log output by either party can be matched using the parameters in the log template.

[0061] According to an embodiment of the present disclosure, obtaining the similarity can be achieved through a machine learning model to calculate the semantic similarity between the second log and the log template. For example, deep learning models such as DSSM (Deep Structured Semantic Models), CNN (convolutional latent semantic model)-DSSM, and LSTM (Long-Short-Term Memory)-DSSM are used to calculate the semantic similarity.

[0062] In operation S240, for each second log, when the similarity meets the first preset condition, it is confirmed that the category of the second log is the same as that of the log template, where the category of the log template includes an automated category or a manual category.

[0063] The automated category is used to determine the logs output by automated test operations. The manual category is used to determine the logs output by manual operations. Thus, without affecting the normal operation of the application function, the separation of automated logs and manual online transaction logs is achieved, facilitating subsequent troubleshooting of transaction information.

[0064] According to an embodiment of the present disclosure, based on the dimension of the business scenario, N second logs are obtained from the first log and each second log is classified. Using the similarity between each second log and the log template as the classification basis, when the similarity meets the first preset condition, it is confirmed that the category of the second log is the same as that of the log template. The content in the first log can be accurately classified. Whether there are problems in transactions during automated testing or manual testing, the log content of the corresponding transaction can be accurately located through the logs of the corresponding category, improving the testing efficiency.

[0065] Figure 3 Schematically shows a flowchart of obtaining the second log in operation S220 according to an embodiment of the present disclosure.

[0066] As Figure 3 shown, obtaining N second logs based on the content of the first log in operation S220 may include operations S310 to S320.

[0067] In operation S310, the content of the first log is rewritten to obtain a third log cluster, where the third log cluster includes S third logs, and S is an integer greater than or equal to 1.

[0068] Rewriting means copying the content of the first log and writing it into the third log. For the convenience of log management, it can be written into one or more third logs to form a third log cluster. The content of the first log is included in this third log cluster.

[0069] In operation S320, N second logs are obtained according to the content of the third log cluster.

[0070] According to an embodiment of the present disclosure, the role of the third log cluster is a backup relative to the first log. Operating to obtain the second log based on the third log cluster can avoid the situation where damage may occur due to directly operating on the first log.

[0071] Figure 4 Schematically shows a flowchart of obtaining the second log in operation S320 according to another embodiment of the present disclosure.

[0072] As Figure 4 shown, obtaining N second logs according to the content of the third log cluster in operation S320 includes obtaining each second log, which may include operations S410 to S440.

[0073] In operation S410, a first start identifier is obtained in the third log, where the first start identifier may be any start identifier in the third log.

[0074] In operation S420, a first thread symbol is obtained according to the first start identifier, where the first start identifier and the first thread symbol are in the first log information in the third log.

[0075] According to an embodiment of the present disclosure, the second log is obtained in units of each third log, and each third log may include log information of multiple transactions. The trigger of each transaction will first output log information such as start identifiers and IP addresses in the log. The start of the log information of a transaction can be judged by the start identifier (such as begin). Whenever the begin start symbol is encountered, the fields in a predetermined order of the log information of this log can be obtained to get the first thread symbol, such as the 8 fields of "000000c0".

[0076] In operation S430, starting from the first log information, at least one log information including the first thread symbol is sequentially obtained.

[0077] Each log information will include a thread symbol field. Starting from the first log information, the thread symbols in each log information are sequentially obtained in chronological order. Among them, since different threads are started in chronological order, the thread symbols included in two adjacent log information may be different. Therefore, if the log information includes the first thread symbol, it is directly obtained; if not, the log information is discarded.

[0078] In operation S440, at least one log information is written into the second log, where the last log information in the at least one log information includes a first end identifier.

[0079] Because multi-threading records logs simultaneously, for example, the log information of "000000de" may be adjacent to that of "000000c0". Search backward and record each log information of the thread symbol "000000c0" into the second log, which can be named with the thread symbol. Until a log information that includes both the first thread symbol and the first end identifier (such as end) is encountered. In some embodiments, after determining end, it can also be determined that the thread symbols of the subsequent 10 (only for example) log information are all inconsistent with the first thread symbol, then it is determined that the second log extracted according to the first thread symbol corresponds to a business transaction.

[0080] According to an embodiment of the present disclosure, obtaining the second log at the granularity of each transaction can accurately classify the transaction logs in automated testing or manual testing. Moreover, by using the first start identifier and the first end identifier to determine the log information of a transaction, comprehensive acquisition can be achieved, avoiding the situation where problems cannot be detected in time due to omission when locating the log information of the transaction.

[0081] According to an embodiment of the present disclosure, for each second log, if it is the same as the category of the log template, it is written into the fourth log. If it is different from the category of the log template, it is written into the fifth log.

[0082] The fourth log can be called a routine log, which is split from the third log and records the application log information generated by automated test execution or the application log information generated by manual test execution. The fifth log can be called a new business log, and the content in this log is opposite to the log category in the fourth log. The content of the fourth log and the fifth log is the entire content of the third log cluster. In some embodiments, the fourth log and the fifth log can also be generated in the form of log clusters, that is, each includes one or more fourth logs and fifth logs.

[0083] According to an embodiment of the present disclosure, the fourth log and the fifth log are log information of different categories, and the content therein is arranged in the order of the thread symbol. When locating the log information of a certain transaction according to the thread symbol, it can be conveniently achieved by using the fourth log and the fifth log.

[0084] Figure 5 Schematically shows a flowchart of obtaining the sixth log according to an embodiment of the present disclosure.

[0085] As Figure 5 shown, the log processing method of this embodiment may further include operation S510 to operation S520.

[0086] In operation S510, match the fourth log with the first log.

[0087] In operation S520, delete the content identical to the fourth log from the first log to obtain the sixth log.

[0088] According to an embodiment of the present disclosure, the sixth log may be referred to as the "new application log", which includes the content remaining after the fourth log - the "routinized log" is removed from the first log. If the "routinized log" contains automated test log information, the sixth log records the application log information generated by manual business transactions and other modules during the execution of automated cases. The content in this log is arranged in chronological order and can provide query logs for manual online business transactions.

[0089] It should be noted that the sixth log can be obtained after all N second logs are classified. It can also be obtained after each second log is classified. If written to the sixth log, the corresponding content in the first log can be deleted.

[0090] To more easily understand the process of obtaining the second log to the sixth log based on the first log, the following is Figure 6 described again.

[0091] Figure 6 Schematically shows a flowchart of a log processing method according to another embodiment of the present disclosure.

[0092] As Figure 6 shown, the log processing method of this embodiment may include operation S610 to operation S680.

[0093] In operation S610, obtain the first log.

[0094] In operation S620, obtain the third log, and operation S310 can be referred to.

[0095] In operation S630, obtain the second log based on the third log, and operation S320, operation S410 to operation S420 can be referred to.

[0096] In operation S640, match the second log with the log template to obtain the similarity between each second log and the log template, and classify the second log. Operation S230, operation S240 can be referred to.

[0097] In operation S650, if the category is the same as the log template, write it to the fourth log.

[0098] In operation S660, if the category is different from the log template, write it to the fifth log.

[0099] In operation S670, obtain the sixth log according to the fourth log and the first log. Operation S510 to operation S520 can be referred to.

[0100] Figure 7Schematically shown is a flowchart of obtaining a third log cluster in operation S310 according to an embodiment of the present disclosure.

[0101] As Figure 7 shown, in operation S310, copying the content of the first log to obtain the third log cluster may include operations S710 to S720.

[0102] In operation S710, monitor the data volume of the first log.

[0103] In operation S720, when the growth rate of the data volume meets the second preset condition, copy the content of the first log, where the second preset condition includes that the growth rate of the data volume is greater than or equal to the first preset threshold.

[0104] The data volume of the first log may be the file size of the first log. By detecting the file size of the first log, after the log growth rate exceeds the first preset threshold (for example, 100 KB / second), start copying. For example, the number of transactions in automated testing is much larger than that in manual testing. After starting automated testing, automatic monitoring can be achieved through the growth rate of the data volume.

[0105] Figure 8 Schematically shown is a flowchart of obtaining a third log cluster in operation S310 according to another embodiment of the present disclosure.

[0106] On the basis of including S710 to operation S720, as Figure 8 shown, this embodiment may further include operations S810 to S820.

[0107] In operation S810, based on the timestamp when the growth rate of the data volume meets the second preset condition, write the log information generated after this timestamp into the i-th third log, where i is an integer greater than or equal to 1 and less than or equal to S.

[0108] Exemplarily, record the current time node as the timestamp, and at the same time generate an automated log cluster, and copy the content written into the application log to the third log in real time.

[0109] In operation S820, when the data volume of the i-th third log is greater than or equal to the second preset threshold, write the log information generated after the timestamp and not yet copied into the (i + 1)-th third log.

[0110] Exemplarily, each new log generated from the "application log" is replicated in real time. The third log has a naming rule of automation_YYYYMMDDHHMMSS_X.log (for example only). YYYYMMDDHHMMSS is the aforementioned timestamp, where YYYY represents the year, MM represents the month, DD represents the day, HH represents the hour, MM represents the minute, SS represents the second, and X represents the serial number. Specifically, counting starts from 1, the date is reused from the previous one, and the serial number needs to be incremented. By default, a new third log will be added every time the automation log exceeds 2M (i.e., the second preset threshold, for example only), and no operation will be performed on the "application log" itself. For example, during the execution of the same automation case, the timestamp will not change, and the serial number will be incremented. The "third log cluster" includes the summary of the third logs composed of the same timestamp and different serial numbers, in the following form:

[0111] automation_20210920164005_1.log;

[0112] automation_20210920164005_2.log;

[0113] automation_20210920164005_3.log; ......

[0115] automation_20210920164005_N.log.

[0116] Figure 9 Schematically shows a flowchart for stopping replicating the first log according to an embodiment of the present disclosure.

[0117] As Figure 9 shown, stopping replicating the first log may include operation S910 to operation S920.

[0118] In operation S910, monitor the number of third logs in the third log cluster.

[0119] In operation S920, when the growth rate of the quantity meets the third preset condition, stop replicating the content of the first log, where the second preset condition includes that the growth rate of the quantity is less than the second preset threshold.

[0120] Exemplarily, during the daily business test process, after starting the automation case test, when it is detected that the number of the first logs exceeds the first preset threshold, replication starts. And after 10 seconds (for example only), the detection of the data volume size is paused, and the growth rate of the number of the third logs is detected instead.

[0121] Exemplarily, after closing the automated case test, it is detected that the growth rate of the number of the third logs is less than the second preset threshold. After 10 seconds (for example only), the replication is stopped and the detection of the growth rate of the number of the third logs is paused, and it is restored to detecting the file size of the first log.

[0122] According to an embodiment of the present disclosure, there may be multiple first logs. Therefore, when a first log file reaches the capacity limit, a new first log may be generated. In this case, it is meaningless to detect the growth rate of the file size of the original first log. Therefore, the start and stop monitoring of the automated case test is achieved by detecting the growth rate of the number of the third logs. It should be understood that in the case where the number of manual tests is much larger than that of automated tests, it is also applicable to the monitoring of manual tests.

[0123] Figure 10 Schematically shows a flowchart of obtaining the similarity in operation S230 according to an embodiment of the present disclosure.

[0124] As Figure 10 shown, obtaining the similarity between each second log and the log template in operation S230 may include operations S1010 to S1020.

[0125] In operation S1010, the buried point information in each second log is matched with the preset buried point information of the log template.

[0126] In operation S1020, the similarity is obtained based on the number of successful matches.

[0127] Exemplarily, since the log information is output in a standard format, the buried point information in the second log can be matched with the preset buried point information of the log template based on the string pattern matching algorithm (KMP algorithm, for example only). The KMP algorithm can consider the paragraph format in the second log and the log template, as well as the string information for matching.

[0128] The buried point information may include one or more keywords in the log information, such as keywords information like the client IP value (such as "83.25.209.23"), product information (such as "Ruyi Life III-D"), etc. If at least one keyword in the second log matches that in the log template, it is considered that the content similarity is high. The number of successful matches can be set according to the actual situation, and the present disclosure does not limit it.

[0129] Figure 11 Schematically shows a flowchart of obtaining the log template according to an embodiment of the present disclosure.

[0130] As Figure 11 shown, before performing operation S230, obtaining the log template in this embodiment may include operations S1110 to S1120.

[0131] In operation S1110, an automated test operation is performed alone, or a manual test operation is performed alone to generate a seventh log.

[0132] In operation S1120, at least one preset buried point information is written into the seventh log to obtain a log template.

[0133] Taking the separate execution of the automated test operation as an example, without the intervention of manual testing, some automated test cases are executed to generate a seventh log. Its function is to obtain the log format output by the automated test. And on this basis, buried points are set according to keywords such as the user source address IP (client), the identifier of the user using, and the information of the product under test, so as to obtain a log template. The log template can dynamically update keywords according to requirements to classify flexibly and accurately. The steps of obtaining the log template by separately executing the manual test operation are the same as those above and will not be elaborated here.

[0134] Figure 12 A flowchart of a log processing method according to another embodiment of the present disclosure is schematically shown.

[0135] As Figure 12 shown, the log processing method of this embodiment may include operations S210 to S240. Before obtaining the similarity between each second log and the log template, operations S1210 to S1240 may also be included.

[0136] In operation S1210, at least one first address information and the category of each first address information are stored, where the first address information is used to mark the client address where the test operation is executed.

[0137] Exemplarily, an IP list (that is, at least one first address information) may be stored in advance, and the category of each IP corresponds to whether the client executes an automated test or a manual test. For example, if all the IPs in the IP list are client addresses for executing automated tests, they are all of the automated category. When an IP in the first log is detected, it can also be used as a condition to trigger the overwriting of the first log.

[0138] In operation S210, a first log generated in response to a test operation is obtained, where the test operation includes an automated test operation and a manual test operation.

[0139] In operation S220, based on the content of the first log, N second logs are obtained, where each second log includes log information generated by testing a first service scenario, and the first service scenario is any one of M service scenarios, and N and M are integers greater than or equal to 1 respectively.

[0140] In operation S1220, it is determined whether there is second address information in the i-th second log, that is, it is determined whether there is an IP address in a certain log message. If so, operation S1230 is executed. If not, operation S230 is executed. The i-th second log can be any second log, and i is an integer greater than or equal to 1.

[0141] In operation S1230, it is determined whether the second address information matches any first address information. If so, operation S1240 is executed. If not, operation S230 is executed.

[0142] For example, using the second address information as the search term, a search is performed in the pre-stored IP list. If a matching IP address is retrieved, the judgment result is considered to be yes.

[0143] In operation S1240, for each second log, in the case where there is second address information, if the second address information matches any first address information, it is confirmed that the second log is of the same category as the first address information that is matched.

[0144] Since the category of each IP address is marked in the pre-stored IP list, if this IP address appears, it indicates that the log category is the same as the category of this IP address.

[0145] In operation S230, the similarity between each second log and the log template is obtained, where the log template includes logs generated by automated test operations or manual test operations.

[0146] In operation S240, for each second log, in the case where the similarity meets the first preset condition, it is confirmed that the second log is of the same category as the log template, where the category of the log template includes an automated category or a manual category.

[0147] It should be noted that operation S1210 can be executed simultaneously with any one of operations S210 or S220, or can be executed before any one of them. The present disclosure does not limit its specific execution order.

[0148] According to an embodiment of the present disclosure, by determining whether the second address information matches the first address information to classify the second log, the classification speed can be improved.

[0149] Based on the above log processing method, the present disclosure also provides a log processing device. The following will be combined with Figure 13 to describe the device in detail.

[0150] Figure 13 The structural block diagram of the log processing device 1300 according to an embodiment of the present disclosure is schematically shown.

[0151] As Figure 13As shown, the log processing device 1300 of this embodiment includes a first acquisition module 1310, a second acquisition module 1320, a log cleaning module 1330, and a log classification module 1340.

[0152] The first acquisition module 1310 may perform operation S210 to acquire a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation.

[0153] The second acquisition module 1320 may perform operation S220 to acquire N second logs based on the content of the first log, where each second log includes log information generated by testing a first business scenario, and the first business scenario is any one of M business scenarios, and N and M are integers greater than or equal to 1 respectively.

[0154] The log cleaning module 1330 may perform operation S230 to obtain the similarity between each second log and a log template, where the log template is obtained based on the log generated by an automated test operation or a manual test operation.

[0155] The log classification module 1340 may perform operation S240 to confirm that, for each second log, when the similarity meets a first preset condition, the category of the second log is the same as that of the log template, where the category of the log template includes an automated category or a manual category.

[0156] The log processing device 1300 may further include an automated access startup module. On the one hand, the automated access startup module may be used to perform operation S1210 to store whitelist IP information. On the other hand, the automated access startup module may perform operations S710 to S720. On yet another hand, the automated access startup module may perform operations S910 to S920.

[0157] The second acquisition module 1320 may further include a log rewriting module. The log rewriting module may, in response to a startup instruction sent by the automated access startup module, perform operation S310, operations S810 to S820. It may also stop the rewriting operation in response to a stop instruction sent by the automated access startup module.

[0158] The log cleaning module 1330 may further perform operation S320, operations S410 to S440, and operations S1010 to S1020, and may also be used to, for each second log, write it into a fourth log if its category is the same as that of the log template, and write it into a fifth log if its category is different from that of the log template.

[0159] The log processing device 1300 may further include a log template module. The log template module may be used to perform operations S1110 to S1120.

[0160] The log processing device 1300 may further include a log rewriting module. The log rewriting module may be used to execute operations S510 to S520.

[0161] The log processing device 1300 may further include an information display module. The information display module is used for front-end display and has input and output functions at the same time. It is possible to input: the whitelist information of the automated access start module, the threshold information of the automated access start module, the start or stop instruction of the automated access start module, the preset buried point information of the log template module, etc. It is possible to output: the whitelist information of the automated access start module, the log template of the log template module, the new service log download of the log cleaning module, the routine log cluster download, the automated log cluster download, etc.

[0162] According to an embodiment of the present disclosure, the log processing device 1300 can be applied to the scenario where manual online transaction logs and automated logs are mixed, realizing log classification and rewriting during enterprise-level automated case testing, and can be adapted to a variety of middleware systems. On the one hand, the logs generated by automated testing are separated from the logs generated by normal manual online business transactions, facilitating business implementers to quickly and accurately query the online transaction logs. On the other hand, it avoids the scenario where logs with a small quantity, such as normal manual transaction logs, are brushed off and cannot be retrieved due to excessive full-volume logs. On the other hand, it supports personalized setting of relevant characters as preset buried point information, making the screening conditions more complete and the applicable scenarios wider. Finally, it supports multiple updates of the log template, continuously supplementing and updating the screening conditions, and has stronger practical timeliness.

[0163] It should be noted that the implementation manners, the technical problems solved, the functions achieved, and the technical effects achieved by each module / unit / sub-unit, etc. in some embodiments of the device are the same as or similar to those of the corresponding steps in some embodiments of the method, and will not be elaborated here.

[0164] According to an embodiment of the present disclosure, any multiple of the first obtaining module 1310, the second obtaining module 1320, the log cleaning module 1330, and the log classification module 1340 may be combined and implemented in one module, or any one of them may be split into multiple modules. Or, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module.

[0165] According to an embodiment of the present disclosure, at least one of the first acquisition module 1310, the second acquisition module 1320, the log cleaning module 1330, and the log classification module 1340 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system in a package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable means such as hardware or firmware for integrating or packaging circuits, or can be implemented in any one of the three implementation manners of software, hardware, and firmware, or in any appropriate combination of several of them. Alternatively, at least one of the first acquisition module 1310, the second acquisition module 1320, the log cleaning module 1330, and the log classification module 1340 can be at least partially implemented as a computer program module, and when the computer program module runs, it can execute corresponding functions.

[0166] Figure 14 FIG. schematically shows a block diagram of an electronic device suitable for implementing a log processing method according to an embodiment of the present disclosure.

[0167] As Figure 14 shown, the electronic device 1400 according to an embodiment of the present disclosure includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1402 or a program loaded from a storage section 1408 into a random access memory (RAM) 1403. The processor 1401 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 1401 can also include on-board memory for caching purposes. The processor 1401 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0168] In the RAM 1403, various programs and data required for the operation of the electronic device 1400 are stored. The processor 1401, the ROM 1402, and the RAM 1403 are connected to each other through a bus 1404. The processor 1401 performs various operations of the method flow according to an embodiment of the present disclosure by executing the programs in the ROM 1402 and / or the RAM 1403. It should be noted that the program can also be stored in one or more memories other than the ROM 1402 and the RAM 1403. The processor 1401 can also perform various operations of the method flow according to an embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0169] According to an embodiment of the present disclosure, the electronic device 1400 may further include an input / output (I / O) interface 1405, and the input / output (I / O) interface 1405 is also connected to the bus 1404. The electronic device 1400 may further include one or more of the following components connected to the I / O interface 1405: an input part 1406 including a keyboard, a mouse, etc.; an output part 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage part 1408 including a hard disk, etc.; and a communication part 1409 including a network interface card such as a LAN card, a modem, etc. The communication part 1409 performs communication processing via a network such as the Internet. The drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed so that a computer program read from it can be installed into the storage part 1408 as needed.

[0170] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0171] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, device, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the above-described ROM 1402 and / or RAM 1403 and / or one or more memories other than ROM 1402 and RAM 1403.

[0172] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the method provided by the embodiments of the present disclosure.

[0173] When the computer program is executed by the processor 1401, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0174] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of signals on a network medium, and be downloaded and installed through the communication part 1409, and / or be installed from the removable medium 1411. The program code included in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0175] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or be installed from the removable medium 1411. When the computer program is executed by the processor 1401, the above functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0176] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include but are not limited to, such as Java, C++, python, the "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0177] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A log processing method, comprising: Obtaining a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation; Based on the content of the first log, obtaining N second logs, where each second log includes log information generated by testing a first business scenario, and the first business scenario is any one of M business scenarios, and N and M are integers greater than or equal to 1 respectively; Obtaining the similarity between each second log and a log template, where the log template is obtained based on logs generated by an automated test operation or a manual test operation; For each second log, when the similarity meets a first preset condition, confirming that the second log is of the same category as the log template, where the category of the log template includes an automated category or a manual category; Wherein, the obtaining N second logs based on the content of the first log includes: Rewriting the content of the first log to obtain a third log cluster, where the third log cluster includes S third logs, and S is an integer greater than or equal to 1; Based on the content of the third log cluster, obtaining N of the second logs; Wherein, the rewriting the content of the first log to obtain a third log cluster includes: Monitoring the data volume of the first log; When the growth rate of the data volume meets a second preset condition, rewriting the content of the first log, where the second preset condition includes that the growth rate of the data volume is greater than or equal to a first preset threshold.

2. The method according to claim 1, wherein The obtaining N of the second logs based on the content of the third log cluster includes obtaining each second log, specifically including: Obtaining a first start identifier in the third log, where the first start identifier is any start identifier in the third log; Obtaining a first thread symbol according to the first start identifier, where the first start identifier and the first thread symbol are in the first log information in the third log; Starting from the first log information, sequentially obtaining at least one log information including the first thread symbol; Writing the at least one log information into the second log, where the last log information in the at least one log information includes a first end identifier.

3. The method according to claim 2, wherein, Further comprising: for each second log, If it is of the same category as the log template, writing it into a fourth log; If it is not of the same category as the log template, writing it into a fifth log.

4. The method according to claim 3, wherein, Further comprising: Matching the fourth log with the first log; Deleting the content identical to the fourth log from the first log to obtain a sixth log.

5. The method according to claim 1, wherein The rewriting the content of the first log to obtain a third log cluster further includes: Based on the timestamp when the growth rate of the data volume meets the second preset condition, writing the log information generated after the timestamp into the i-th third log, where i is an integer greater than or equal to 1 and less than or equal to S; When the data volume of the i-th third log is greater than or equal to a second preset threshold, writing the log information generated after the timestamp and not rewritten into the (i + 1)-th third log.

6. The method according to claim 5, wherein Further comprising: Monitor the number of third logs in the third log cluster; When the growth rate of the number meets the third preset condition, stop overwriting the content of the first log, where the second preset condition includes that the growth rate of the number is less than the second preset threshold.

7. The method according to claim 1, wherein The obtaining the similarity between each second log and the log template includes: Match the buried point information in each second log with the preset buried point information of the log template; Obtain the similarity based on the number of successful matches.

8. The method according to claim 7, wherein Before obtaining the similarity between each second log and the log template, it further includes obtaining the log template, specifically including: Individually execute the automated test operation or individually execute the manual test operation to generate a seventh log; Write at least one of the preset buried point information into the seventh log to obtain the log template.

9. The method according to claim 1, wherein, Includes: Store at least one first address information and the category of each first address information, where the first address information is used to mark the client address where the test operation is executed; Before obtaining the similarity between each second log and the log template, the method further includes: For each second log, when there is second address information, if the second address information matches any one of the first address information, confirm that the second log is of the same category as the first address information it matches.

10. A log processing device, including: A first obtaining module, configured to obtain a first log generated in response to a test operation, where the test operation includes an automated test operation and a manual test operation; A second obtaining module, configured to obtain N second logs based on the content of the first log, where each second log includes log information generated by testing a first service scenario, and the first service scenario is any one of M service scenarios, and N and M are integers greater than or equal to 1; A log cleaning module, configured to obtain the similarity between each second log and the log template, where the log template is obtained based on the log generated by the automated test operation or the manual test operation; A log classification module, configured to, for each second log, when the similarity meets the first preset condition, confirm that the second log is of the same category as the log template, where the category of the log template includes an automated category or a manual category; Wherein, the obtaining N second logs based on the content of the first log includes: Overwrite the content of the first log to obtain a third log cluster, where the third log cluster includes S third logs, and S is an integer greater than or equal to 1; Obtain N of the second logs according to the content of the third log cluster; Wherein, the overwriting the content of the first log to obtain a third log cluster includes: Monitor the data volume of the first log; When the growth rate of the data volume meets the second preset condition, overwrite the content of the first log, where the second preset condition includes that the growth rate of the data volume is greater than or equal to the first preset threshold.

11. An electronic device, including: One or more processors; A storage device for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having executable instructions stored thereon, which when executed by a processor cause the processor to execute the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, which when executed by a processor implements the method according to any one of claims 1 to 9.

Images