Transaction log collection method, apparatus and system

Abstract

The application provides a transaction log collection method, device and system. The transaction log collection method comprises the following steps: putting log files of each application server into a cache queue; determining a time difference according to a tail transaction time and a head message transaction time; determining a log output time according to the time difference and the head message transaction time; sorting the log files in the cache queue according to transaction times in the log files, and putting the sorted log files into an output message queue according to the log output time. The application can effectively deal with a large number of logs generated in a short time, ensure that the collected logs are complete and orderly, avoid transaction failure, and further improve the playback efficiency and effect of a subsequent follow-up playback device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data data analysis, in particular to a transaction log collection method, device and system. BACKGROUND

[0002] Figure 1 is a transaction log generation schematic diagram. As shown in Figure 1 , a user initiates a request to the backend through various channel ends provided by the bank, such as opening an account, lending, and the like, and the server records each request information (Req, i.e., request message) and response information (Rep, i.e., response message) into the log. The transaction playback system reproduces the transaction according to the log. The log message collection system is used to collect the message information and send it to the Kafka message queue for the accounting playback system to consume.

[0003] The bank system is a distributed system, and the log is distributed on each application server (AP) and updated in real time. The collection agent on the application server has a scanning interval when scanning the log. If the collected log file has a strong requirement for time sequence, problems may occur.

[0004] Figure 2 is a transaction time sequence access schematic diagram. The transaction refers to a request initiated by the bank system from the channel end, as shown in Figure 2 , in a distributed system, the transaction is evenly sent to each application server AP through load balancing. Assuming that the following transaction call occurs: request (Req) 1-> request 2-> request 3, the transaction is distributed on different application servers and the operation interval is less than 1s, and the scanning interval of the application server is 1s. If the scanning of the application server 1 just ends at this time, it needs to wait for 1s before starting to scan, at this time, the message information of the application server 2 may be recorded in the message queue in advance, and similarly, the request message of the application server 3 may also be collected before request 1 and request 2. When the accounting playback system plays back according to the received message, the time sequence will be chaotic, causing the transaction to fail.

[0005] Figure 3 is a log collection system of the prior art. As shown in Figure 3 , the collection agent agent is deployed on each application server (AP) to scan the system log. When collecting the log, the agent directly stores the collected log data into the target storage system according to the data collection order, and does not process the collected log data in time sequence. In the accounting consumption system, it is necessary to ensure that the execution order of the message is consistent with the original execution order, so it is necessary to process the collected log data in time sequence. SUMMARY

[0006] The main purpose of the embodiment of the present application is to provide a transaction log collection method, device and system to ensure the completeness and order of the collected logs and avoid transaction failure.

[0007] To achieve the above purpose, the embodiment of the present application provides a transaction log collection method, comprising:

[0008] Putting the log files of each application server into a cache queue;

[0009] Determining a time difference according to the tail transaction time and the head message transaction time;

[0010] Determining a log output time according to the time difference and the head message transaction time;

[0011] Sorting the log files in the cache queue according to the transaction time in the log files, and putting the sorted log files into an output message queue according to the log output time.

[0012] In one embodiment, putting the sorted log files into the output message queue according to the log output time comprises:

[0013] Determining a target log file according to the log output time and the transaction time of each log file;

[0014] Putting the sorted target log file into the output message queue.

[0015] In one embodiment, putting the log files of each application server into the cache queue comprises:

[0016] Putting the log files of each application server into a corresponding message queue;

[0017] Putting the log files in each message queue into the cache queue in parallel.

[0018] In one embodiment, the method further comprises:

[0019] Determining the tail transaction time according to the transaction time of the last log file put into the cache queue by each message queue.

[0020] In one embodiment, the method further comprises:

[0021] Determining the transaction time of the head log file of the cache queue as the head message transaction time; or

[0022] Determining the head message transaction time according to the transaction time of the first log file put into the cache queue by each message queue.

[0023] In one embodiment, the method further comprises:

[0024] Adjusting the number of log files put into the cache queue in parallel according to the time difference and the log scanning interval.

[0025] The embodiment of the present application also provides a transaction log collection device, comprising:

[0026] a cache queue module, which is used for putting log files of each application server into a cache queue;

[0027] a time difference determination module, which is used for determining a time difference according to a tail transaction time and a head message transaction time;

[0028] a log output time determination module, which is used for determining a log output time according to the time difference and the head message transaction time;

[0029] a sorting module, which is used for sorting log files in the cache queue according to transaction times in the log files, and putting the sorted log files into an output message queue according to the log output time.

[0030] In one embodiment, the sorting module comprises:

[0031] a target log file unit, which is used for determining a target log file according to the log output time and transaction times of each log file;

[0032] an output message queue unit, which is used for putting the sorted target log file into the output message queue.

[0033] In one embodiment, the cache queue module comprises:

[0034] a message queue unit, which is used for putting log files of each application server into a corresponding message queue;

[0035] a cache queue unit, which is used for putting log files in each message queue into the cache queue in parallel.

[0036] In one embodiment, the device further comprises:

[0037] a tail transaction time determination module, which is used for determining a tail transaction time according to a transaction time of a last log file put into the cache queue by each message queue.

[0038] In one embodiment, the device further comprises:

[0039] a head message transaction time determination module, which is used for determining a transaction time of a head log file of the cache queue as a head message transaction time; or

[0040] determining the head message transaction time according to a transaction time of a first log file put into the cache queue by each message queue.

[0041] In one embodiment, the device further comprises:

[0042] The cache file adjusting module is configured to adjust the number of log files placed in the cache queue in parallel according to the time difference and the log scanning interval.

[0043] The embodiment of the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements the steps of the transaction log collection method when running the computer program.

[0044] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the processor implements the steps of the transaction log collection method when running the computer program.

[0045] The embodiment of the present application also provides a computer program product, which comprises computer program / instructions, and the computer program / instructions implement the steps of the transaction log collection method when running by the processor.

[0046] The embodiment of the present application also provides a transaction log collection system, which comprises:

[0047] The transaction log collection device as described above;

[0048] The application servers are configured to generate log files according to transaction times and transaction messages, and send the log files to the transaction log collection device according to a log scanning interval;

[0049] The account playback device is configured to consume the sorted log files obtained from the output message queue in the transaction log collection device.

[0050] The transaction log collection method, device and system of the embodiment of the present application firstly place the log files of the application servers in the cache queue, then determine the time difference according to the tail transaction time and the head message transaction time to further determine the log output time, and place the log files sorted according to the transaction times in the log files in the output message queue according to the log output time, so that the short-time large amount of generated logs can be effectively coped with, the collected logs can be ensured to be complete and orderly, the transaction failure can be avoided, and then the playback efficiency and playback effect of the subsequent account playback device can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 is a transaction log generation schematic diagram;

[0053] Figure 2 is a transaction time access schematic diagram;

[0054] Figure 3 is a prior art log collection system;

[0055] Figure 4 is a flow chart of a transaction log collection method in an embodiment of the present application;

[0056] Figure 5 is a flow chart of a transaction log collection method in another embodiment of the present application;

[0057] Figure 6 is a schematic diagram of an NTP server in an embodiment of the present application;

[0058] Figure 7 is a flow chart of S101 in an embodiment of the present application;

[0059] Figure 8 is a flow chart of a log file reading exception in an embodiment of the present application;

[0060] Figure 9 is a structural block diagram of a transaction log collection device in an embodiment of the present application;

[0061] Figure 10 is a structural block diagram of a computer device in an embodiment of the present application;

[0062] Figure 11 is a schematic diagram of a transaction log collection system in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0064] Those skilled in the art will appreciate that the embodiments of the present application can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can be embodied in the form of entire hardware, entire software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0065] In view of the transaction failure caused by the lack of time sequence processing on collected log data in the prior art, the embodiments of the present application provide a transaction log collection method to ensure the order of logs while ensuring the integrity of logs by using cache and multi-level message queue output, thereby avoiding transaction failure. The present application will be described in detail below with reference to the drawings.

[0066] Figure 4 is a flow chart of the transaction log collection method in an embodiment of the present application. Figure 5 is a flow chart of the transaction log collection method in another embodiment of the present application. As shown in Figure 4 and Figure 5 , the transaction log collection method comprises:

[0067] S101: placing log files of each application server into a cache queue.

[0068] Before performing S101, the present application further comprises: the present application implements time service through an NTP server (Network Time Protocol, NTP), which can ensure that the time of each application server is consistent, thereby ensuring that the transaction time in the recorded log is accurate, and the sorting output can be based on this as a benchmark. The NTP server is a protocol used to synchronize computer time, which can synchronize the computer to its server or clock source (such as a quartz clock, GPS, etc.), provide high-precision time correction (less than 1 millisecond difference between LAN and standard, tens of milliseconds on WAN), and can prevent malicious protocol attacks through encrypted confirmation.

[0069] Figure 6 is a schematic diagram of the NTP server in an embodiment of the present application. As shown in Figure 6 , the client of the NTP server is configured inside each application server, and the client regularly performs time synchronization operation with the external NTP server through the NTP protocol, so as to ensure the consistency and accuracy of the time of each application server, and further ensure the accuracy of the transaction time in the recorded log file.

[0070] Figure 7 is a flow chart of S101 in an embodiment of the present application. As shown in Figure 7 , S101 comprises:

[0071] S201: placing log files of each application server into a corresponding message queue.

[0072] In specific implementation, each application server writes the transaction time and the transaction message together into the log file according to the received transaction time, which is single-threaded writing and can ensure the time sequence accuracy of writing. The log collection agent inside each application server scans the configured log file at regular intervals, and places the read message information into the designated message queue in single thread, and each application server corresponds to one message queue.

[0073] The operation of placing the log file into the message queue is also a single-threaded operation, which can ensure that the message in each message queue is recorded in time sequence. In order to ensure the timeliness of recording the message, the interval time of scanning the log of each application server can be set to 1s.

[0074] S202: Parallelly put the log files in each message queue into the cache queue.

[0075] In an embodiment, the log files can be continuously read from each message queue to the cache queue by the message processing middleware, and after processing the log files, the log files in the cache queue are output to the final output message queue for consumption by the accounting playback system.

[0076] In specific implementation, the message processing middleware first scans the registered application server services and obtains the corresponding message queues, and then starts multiple threads to read n log files from each message queue in parallel. n is the number of log files parallelly put into the cache queue, and the initial value is the average of the daily transaction volume, which can be automatically adjusted according to the system running condition subsequently.

[0077] Figure 8 is a flowchart of log file reading exception in the embodiment of the application. As shown in Figure 8 If the thread fails to read the log file, the message processing middleware first determines whether a probe transaction needs to be sent to the application server. When the preliminary judgment of the reading failure reason is network exception, the corresponding application server can be directly marked as abnormal without sending a probe transaction to the application server; when the reading failure reason cannot be obtained through preliminary judgment, a probe transaction is sent to the application server.

[0078] After receiving the probe transaction information, the application server returns the transaction execution result to the message processing middleware. When the transaction execution result is successful, it is determined that the current is a normal no-transaction scenario, and an empty message list is returned; when the transaction execution result is failed, the probe transaction is continuously sent to the application server, and when the repeated sending fails for a preset number of times, it is determined that the corresponding application server has stopped receiving services, the application server is marked as abnormal, and an empty message list is directly returned, and the application server list is updated, and the application server marked as abnormal is deleted from the application server list to be scanned.

[0079] After marking the application server as abnormal, the message processing middleware of the application also starts a background monitoring sub-thread, and polls whether the deleted application server is normal according to a preset configuration strategy. When the application server is normal, the application server list is updated, and the log file scanning of the corresponding application server is restarted.

[0080] S102: Determine the time difference according to the tail transaction time and the head message transaction time.

[0081] The time difference t is the difference between the tail transaction time and the head message transaction time.

[0082] In an embodiment, before S102 is executed, the following steps are further included:

[0083] 1. The tail transaction time is determined according to the transaction time of the last log file put into the cache queue by each message queue.

[0084] The tail transaction time is the maximum value of the transaction time of the last log file put into the cache queue by each message queue. For example, if there are three message queues, the transaction time of the last log file put into the cache queue by the first message queue is 10:25:12, the transaction time of the last log file put into the cache queue by the second message queue is 10:27:23, and the transaction time of the last log file put into the cache queue by the third message queue is 10:26:11, then the tail transaction time is 10:27:23.

[0085] 2. The transaction time of the head log file of the cache queue is determined as the head message transaction time; or

[0086] The head message transaction time is determined according to the transaction time of the first log file put into the cache queue by each message queue.

[0087] In an embodiment, when there is a log file in the cache queue before the log file in the message queue is put into the cache queue, the transaction time of the first log file in the cache queue is taken as the head message transaction time. When the cache queue is empty before the log file in the message queue is put into the cache queue, the minimum value of the transaction time of the first log file put into the cache queue by each message queue is determined as the head message transaction time.

[0088] For example, if there are three message queues, the transaction time of the first log file put into the cache queue by the first message queue is 10:01:21, the transaction time of the first log file put into the cache queue by the second message queue is 10:02:32, and the transaction time of the first log file put into the cache queue by the third message queue is 10:01:23, when the cache queue is empty before the log file in the message queue is put into the cache queue, the head message transaction time is 10:01:21.

[0089] S103: Determine the log output time according to the time difference and the head message transaction time.

[0090] The log output time is the sum of the head message transaction time and t / 2, and t is the time difference.

[0091] In an embodiment, it further comprises adjusting the number of log files put into the cache queue in parallel according to the time difference and the log scanning interval.

[0092] In a specific implementation, when the time difference t is less than the first preset time threshold, the number of log files placed into the cache queue in parallel is doubled; when the time difference t is greater than the second preset time threshold, the number of log files placed into the cache queue in parallel is halved; when the time difference t is greater than or equal to the first preset time threshold and less than or equal to the second preset time threshold, the number of log files placed into the cache queue in parallel remains unchanged.

[0093] For example, assuming that the number of log files placed into the cache queue in parallel initially is n. When the time difference t is less than the first preset time threshold, n can be adjusted to 2n, and the message processing middleware will next read 2n log files from each message queue in parallel; when the time difference t is greater than the second preset time threshold, n can be adjusted to n / 2, and the message processing middleware will next read n / 2 log files from each message queue in parallel; when the time difference t is greater than or equal to the first preset time threshold and less than or equal to the second preset time threshold, the message processing middleware will next continue to read n log files from each message queue in parallel.

[0094] The first preset time threshold and the second preset time threshold both depend on the log scanning interval Δt, and the first preset time threshold can be set to 2Δt and the second preset time threshold can be set to 4Δt.

[0095] S104: The log files in the cache queue are sorted according to the transaction time in the log files, and the sorted log files are placed into the output message queue according to the log output time.

[0096] In a specific implementation, the log files in the cache queue can be sorted from front to back according to the transaction time in the log files. Assuming that the log generation time is a, the time difference is t, and the log scanning interval is Δt, it can be determined that a+Δt will be scanned and sent to the Kafka message queue. For example, if a=0, t=2, and Δt=1, it can be determined that the log file with the transaction time of 1 will be scanned and sent to the Kafka message queue. Figure 2 For example, at this time, it is only required that request 1 precede request 2, and request 2 precede request 3, and each time the log of t / 2 is output, it can be ensured that the transaction log of t / 2 is in order and there is no pre-transaction.

[0097] In an embodiment, placing the sorted log files into the output message queue according to the log output time comprises:

[0098] Determining a target log file according to the log output time and the transaction time of each log file, and placing the sorted target log file into the output message queue.

[0099] In a specific implementation, the log file with the transaction time located before the log output time is determined as the target log file.

[0100] For example, the head message transaction time is x, the time difference is t, and the log output time is x+t / 2, and the log file with a transaction time before x+t / 2 is put into the output message queue.

[0101] Figure 4 The execution subject of the transaction log collection method shown can be applied to a transaction log collection device of a message processing middleware. Figure 4 As shown in the flow, the transaction log collection method of the embodiment of the present application first puts the log files of each application server into the cache queue, then determines the time difference according to the tail transaction time and the head message transaction time to further determine the log output time, and puts the log files sorted according to the transaction time in the log files into the output message queue according to the log output time, which can effectively deal with a large number of logs generated in a short time, ensure the completeness and order of the collected logs, avoid transaction failure, and further improve the playback efficiency and effect of the subsequent follow-up playback device.

[0102] Based on the same inventive concept, the embodiment of the present application further provides a transaction log collection device, since the principle of the device for solving the problem is similar to the transaction log collection method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0103] Figure 9 is a structural block diagram of the transaction log collection device in the embodiment of the present application. As shown in the figure, Figure 9 The transaction log collection device comprises:

[0104] The cache queue module is configured to put the log files of each application server into the cache queue.

[0105] The time difference determination module is configured to determine the time difference according to the tail transaction time and the head message transaction time.

[0106] The log output time determination module is configured to determine the log output time according to the time difference and the head message transaction time.

[0107] The sorting module is configured to sort the log files in the cache queue according to the transaction time in the log files, and put the sorted log files into the output message queue according to the log output time.

[0108] In one embodiment, the sorting module comprises:

[0109] The target log file unit is configured to determine the target log file according to the log output time and the transaction time of each log file.

[0110] The output message queue unit is configured to put the sorted target log file into the output message queue.

[0111] In one embodiment, the cache queue module comprises:

[0112] a message queue unit configured to put log files of each application server into a corresponding message queue;

[0113] a cache queue unit configured to put log files in each message queue into a cache queue in parallel.

[0114] In one embodiment, the transaction log collection device further comprises:

[0115] a tail transaction time determination module configured to determine a tail transaction time according to a transaction time of a last log file put into the cache queue by each message queue.

[0116] In one embodiment, the transaction log collection device further comprises:

[0117] a head message transaction time determination module configured to determine a transaction time of a head log file of the cache queue as a head message transaction time; or

[0118] determine the head message transaction time according to a transaction time of a first log file put into the cache queue by each message queue.

[0119] In one embodiment, the transaction log collection device further comprises:

[0120] a cache file adjustment module configured to adjust a number of log files put into the cache queue in parallel according to the time difference and a log scanning interval.

[0121] In summary, the transaction log collection device according to the embodiments of the present application first puts log files of each application server into a cache queue, then determines a time difference according to a tail transaction time and a head message transaction time to further determine a log output time, and finally puts log files sorted according to transaction times in the log files into an output message queue according to the log output time, which can effectively deal with a large number of logs generated in a short time, ensure that collected logs are complete and in order, avoid transaction failure, and further improve the playback efficiency and effect of a subsequent follow-up playback device.

[0122] The embodiments of the present application further provide a specific implementation of a computer device capable of implementing all steps in the transaction log collection method according to the embodiments. Figure 10 is a structural block diagram of the computer device according to the embodiments of the present application, referring to Figure 10 , the computer device specifically comprises the following contents:

[0123] a processor 1001 and a memory 1002.

[0124] The processor 1001 is configured to invoke a computer program in the memory 1002, and when the processor executes the computer program, all steps in the transaction log collection method in the above embodiments are implemented, for example, when the processor executes the computer program, the following steps are implemented:

[0125] Putting the log files of each application server into a cache queue;

[0126] Determining a time difference according to the tail transaction time and the head message transaction time;

[0127] Determining a log output time according to the time difference and the head message transaction time;

[0128] Sorting the log files in the cache queue according to the transaction time in the log files, and putting the sorted log files into an output message queue according to the log output time.

[0129] The putting the sorted log files into an output message queue according to the log output time comprises:

[0130] Determining a target log file according to the log output time and the transaction time of each log file;

[0131] Putting the sorted target log file into the output message queue.

[0132] The putting the log files of each application server into a cache queue comprises:

[0133] Putting the log files of each application server into a corresponding message queue;

[0134] Putting the log files in each message queue into the cache queue in parallel.

[0135] The method further comprises:

[0136] Determining the tail transaction time according to the transaction time of the last log file put into the cache queue by each message queue.

[0137] The method further comprises:

[0138] Determining the transaction time of the head log file of the cache queue as the head message transaction time; or

[0139] Determining the head message transaction time according to the transaction time of the first log file put into the cache queue by each message queue.

[0140] The method further comprises:

[0141] Adjusting the number of log files put into the cache queue in parallel according to the time difference and the log scanning interval.

[0142] In summary, the computer device of the embodiment of the present application firstly puts the log files of each application server into the cache queue, then determines the time difference according to the tail transaction time and the head message transaction time to further determine the log output time, and puts the log files sorted according to the transaction time in the log files into the output message queue according to the log output time, so that the log files generated in a short time can be effectively dealt with, the collected log files are guaranteed to be complete and orderly, transaction failure is avoided, and the playback efficiency and playback effect of the subsequent follow-up playback device are improved.

[0143] The embodiment of the present application also provides a computer readable storage medium capable of realizing all steps of the transaction log collection method in the above-mentioned embodiment, and the computer readable storage medium stores a computer program.

[0144] putting the log files of each application server into the cache queue;

[0145] determining the time difference according to the tail transaction time and the head message transaction time;

[0146] determining the log output time according to the time difference and the head message transaction time;

[0147] sorting the log files in the cache queue according to the transaction time in the log files, and putting the sorted log files into the output message queue according to the log output time.

[0148] The step of putting the sorted log files into the output message queue according to the log output time comprises:

[0149] determining the target log file according to the log output time and the transaction time of each log file;

[0150] putting the sorted target log file into the output message queue.

[0151] The step of putting the log files of each application server into the cache queue comprises:

[0152] putting the log files of each application server into the corresponding message queue;

[0153] putting the log files in each message queue into the cache queue in parallel.

[0154] The embodiment of the present application also comprises:

[0155] determining the tail transaction time according to the transaction time of the last log file put into the cache queue by each message queue.

[0156] The embodiment of the present application also comprises:

[0157] determining the transaction time of the head log file in the cache queue as the head message transaction time; or

[0158] determining the head message transaction time according to the transaction time of the first log file put into the cache queue according to each message queue.

[0159] wherein, further comprising:

[0160] adjusting the number of log files put into the cache queue in parallel according to the time difference and the log scanning interval.

[0161] In summary, the computer readable storage medium of the embodiment of the present application first puts the log files of each application server into the cache queue, then determines the time difference according to the tail transaction time and the head message transaction time to further determine the log output time, and puts the log files sorted according to the transaction time in the log files into the output message queue according to the log output time, which can effectively deal with a large number of logs generated in a short time, ensure the completeness and order of the collected logs, avoid transaction failure, and further improve the playback efficiency and effect of the subsequent follow-up playback device.

[0162] The embodiment of the present application also provides a computer program product capable of implementing all steps of the transaction log collection method in the above-mentioned embodiment, wherein the computer program product comprises computer programs / instructions, and when the computer programs / instructions are executed by a processor, all steps of the transaction log collection method in the above-mentioned embodiment are implemented, for example, when the computer programs / instructions are executed by the processor, the following steps are implemented:

[0163] putting the log files of each application server into the cache queue;

[0164] determining the time difference according to the tail transaction time and the head message transaction time;

[0165] determining the log output time according to the time difference and the head message transaction time;

[0166] sorting the log files in the cache queue according to the transaction time in the log files, and putting the sorted log files into the output message queue according to the log output time.

[0167] wherein, putting the sorted log files into the output message queue according to the log output time comprises:

[0168] determining the target log file according to the log output time and the transaction time of each log file;

[0169] putting the sorted target log file into the output message queue.

[0170] wherein, putting the log files of each application server into the cache queue comprises:

[0171] Put the log files of each application server into the corresponding message queue;

[0172] Put the log files in each message queue into the cache queue in parallel.

[0173] Further comprising:

[0174] Determine the tail transaction time according to the transaction time of the last log file put into the cache queue by each message queue.

[0175] Further comprising:

[0176] Determine the head log file of the cache queue as the head message transaction time; or

[0177] Determine the head message transaction time according to the transaction time of the first log file put into the cache queue by each message queue.

[0178] Further comprising:

[0179] Adjust the number of log files put into the cache queue in parallel according to the time difference and the log scanning interval.

[0180] In summary, the computer program product of the embodiment of the present application first puts the log files of each application server into the cache queue, then determines the time difference according to the tail transaction time and the head message transaction time to further determine the log output time, and puts the log files sorted according to the transaction time in the log files into the output message queue according to the log output time, which can effectively deal with a large number of logs generated in a short time, ensure the completeness and order of the collected logs, avoid transaction failure, and further improve the playback efficiency and effect of the subsequent follow-up playback device.

[0181] Based on the same inventive concept, the embodiment of the present application also provides a transaction log collection system, since the principle of the system for solving the problem is similar to the transaction log collection method, therefore the implementation of the system can be referred to the implementation of the method, and the repeated parts will not be described again.

[0182] Figure 11 is a schematic diagram of the transaction log collection system in the embodiment of the present application. As shown in Figure 11 The transaction log collection system comprises:

[0183] The transaction log collection device as described above;

[0184] Each application server is used to generate log files according to the transaction time and the transaction message, and send the log files to the transaction log collection device according to the log scanning interval;

[0185] The follow-up playback device is used to consume the sorted log files obtained from the output message queue in the transaction log collection device.

[0186] As Figure 11 shown, the follow-up playback device is a real-time playback of production transactions follow-up test device, transaction log collection device can be applied to the message processing middleware, including the output message queue and the message queue corresponding to the application server one by one, the message processing middleware includes a cache queue. The application flow of the embodiment of the present application is as follows:

[0187] 1. Each application server generates a log file according to the transaction time and the transaction message, and sends the log file to the corresponding message queue according to the log scanning interval.

[0188] 2. The message processing middleware of the transaction log collection device starts multiple threads to read n log files from each message queue in parallel.

[0189] 3. The message processing middleware of the transaction log collection device determines the tail transaction time according to the transaction time of the last log file put into the cache queue by each message queue. When the log file in the message queue is put into the cache queue, there is already a log file in the cache queue, the message processing middleware takes the transaction time of the first log file in the cache queue as the head message transaction time. When the log file in the message queue is put into the cache queue, the cache queue is empty, the message processing middleware determines the minimum value of the transaction time of the first log file put into the cache queue by each message queue as the head message transaction time.

[0190] 4. The message processing middleware of the transaction log collection device determines the log output time according to the tail transaction time and the head message transaction time, and determines the time difference according to the tail transaction time and the head message transaction time.

[0191] 5. The message processing middleware of the transaction log collection device determines the log output time according to the time difference and the head message transaction time, and sorts the log files in the cache queue according to the transaction time in the log file.

[0192] 6. The message processing middleware of the transaction log collection device determines the target log file according to the log output time and the transaction time of each log file, and puts the sorted target log file into the output message queue.

[0193] 7. The follow-up playback device consumes the sorted log file obtained from the output message queue in the transaction log collection device.

[0194] In summary, the transaction log collection system of the embodiment of the present application firstly puts the log files of each application server into the cache queue, then determines the time difference according to the tail transaction time and the head message transaction time to further determine the log output time, and puts the log files sorted according to the transaction time in the log files into the output message queue according to the log output time, so that the short-time large amount of generated logs can be effectively dealt with, the collected logs can be ensured to be complete and orderly, the transaction failure can be avoided, and the playback efficiency and playback effect of the subsequent follow-up playback device can be improved.

[0195] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the above-described specific embodiments are merely specific embodiments of the present application and are not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0196] Those skilled in the art can also understand that the various illustrative logical blocks, units and steps listed in the embodiments of the present application can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the above various illustrative components, units and steps have been generally described their functions. Whether the functions are realized by hardware or software depends on the specific application and design requirements of the whole system. Those skilled in the art can use various methods to realize the functions for each specific application, but such implementation should not be understood as beyond the scope of protection of the embodiments of the present application.

[0197] The various illustrative logical blocks or units or devices described in the embodiments of the present application can be realized or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination of any of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be realized by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0198] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0199] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

Claims

1. A method for collecting transaction logs, characterized in that, include: Place the log files of each application server into a cache queue; The time difference is determined based on the end-of-day transaction time and the beginning-of-day transaction time. The log output time is determined based on the time difference and the header message transaction time; where the header message transaction time is x, the time difference is t, and the log output time is x+t / 2. Log files with transaction times before x+t / 2 are placed into the output message queue. The log files in the cache queue are sorted according to the transaction time in the log files, and the sorted log files are put into the output message queue according to the log output time. The transaction log collection method also includes: The number of log files placed into the cache queue in parallel is adjusted based on the time difference and log scan interval. Specifically, when the time difference t is less than the first preset time threshold, the number of log files placed into the cache queue in parallel is doubled; when the time difference t is greater than the second preset time threshold, the number of log files placed into the cache queue in parallel is halved; when the time difference t is greater than or equal to the first preset time threshold and less than or equal to the second preset time threshold, the number of log files placed into the cache queue in parallel remains unchanged. Placing the log files of each application server into the cache queue includes: Place the log files of each application server into the corresponding message queue; Log files from each message queue are placed into the cache queue in parallel. The message processing middleware continuously reads log files from various message queues into a cache queue. After processing the log files, the log files in the cache queue are output to the final output message queue. The message processing middleware first scans the registered application server services and obtains the corresponding message queues. Then, it starts multiple threads to read n log files in parallel from each message queue. n is the number of log files that are put into the cache queue in parallel. The initial value is the average daily transaction volume, which can be automatically adjusted according to the system operation. If a thread fails to read the log file, the message processing middleware first determines whether a probe transaction needs to be sent to the application server. If the initial determination is that the read exception is due to a network error, the corresponding application server is directly marked as abnormal, and no probe transaction needs to be sent to the application server. If the reason for the read exception cannot be obtained through the initial determination, a probe transaction is sent to the application server. After receiving the probe transaction information, the application server returns the transaction execution result to the message processing middleware. If the transaction execution result is successful, it is determined that the current scenario is normal with no transactions, and an empty message list is returned. If the transaction execution result is unsuccessful, probe transactions are sent to the application server again. After failing to send the preset number of times, it is determined that the corresponding application server has stopped receiving services, the application server is marked as abnormal, an empty message list is returned directly, the application server list is updated, and the application server marked as abnormal is removed from the list of application servers to be scanned. After marking an application server as abnormal, the message processing middleware will also start a background monitoring thread to poll whether the deleted application servers are normal according to a preset configuration strategy; when the application server is normal, the application server list will be updated and the log file scan of the corresponding application server will be restarted.

2. The transaction log collection method according to claim 1, characterized in that, The process of placing the sorted log files into the output message queue according to the log output time includes: The target log file is determined based on the log output time and the transaction time of each log file. The sorted target log files are placed into the output message queue.

3. The transaction log collection method according to claim 1, characterized in that, Also includes: The tail transaction time is determined based on the transaction time of the last log file placed into the cache queue for each message queue.

4. The transaction log collection method according to claim 1, characterized in that, Also includes: The transaction time of the header log file in the cache queue is determined as the transaction time of the header message; or The header message transaction time is determined based on the transaction time of the first log file placed into the cache queue for each message queue.

5. A transaction log collection device, characterized in that, include: The cache queue module is used to put the log files of each application server into the cache queue; The time difference determination module is used to determine the time difference between the tail transaction time and the head message transaction time; The log output time determination module is used to determine the log output time based on the time difference and the header message transaction time; wherein, the header message transaction time is x, the time difference is t, then the log output time is x+t / 2, and log files with transaction times before x+t / 2 are placed into the output message queue; The sorting module is used to sort the log files in the cache queue according to the transaction time in the log files, and put the sorted log files into the output message queue according to the log output time. The cache file adjustment module is used to adjust the number of log files that are placed into the cache queue in parallel according to the time difference and the log scan interval; Specifically, when the time difference t is less than the first preset time threshold, the number of log files placed into the cache queue in parallel is doubled; when the time difference t is greater than the second preset time threshold, the number of log files placed into the cache queue in parallel is halved; when the time difference t is greater than or equal to the first preset time threshold and less than or equal to the second preset time threshold, the number of log files placed into the cache queue in parallel remains unchanged. The cache queue module includes: The message queue unit is used to put the log files of each application server into the corresponding message queue; The cache queue unit is used to put log files from each message queue into the cache queue in parallel. The message processing middleware continuously reads log files from various message queues into a cache queue. After processing the log files, the log files in the cache queue are output to the final output message queue. The message processing middleware first scans the registered application server services and obtains the corresponding message queues. Then, it starts multiple threads to read n log files in parallel from each message queue. n is the number of log files that are put into the cache queue in parallel. The initial value is the average daily transaction volume, which can be automatically adjusted according to the system operation. If a thread fails to read the log file, the message processing middleware first determines whether a probe transaction needs to be sent to the application server. If the initial determination is that the read exception is due to a network error, the corresponding application server is directly marked as abnormal, and no probe transaction needs to be sent to the application server. If the reason for the read exception cannot be obtained through the initial determination, a probe transaction is sent to the application server. After receiving the probe transaction information, the application server returns the transaction execution result to the message processing middleware. If the transaction execution result is successful, it is determined that the current scenario is normal with no transactions, and an empty message list is returned. If the transaction execution result is unsuccessful, probe transactions are sent to the application server again. After failing to send the preset number of times, it is determined that the corresponding application server has stopped receiving services, the application server is marked as abnormal, an empty message list is returned directly, the application server list is updated, and the application server marked as abnormal is removed from the list of application servers to be scanned. After marking an application server as abnormal, the message processing middleware will also start a background monitoring thread to poll whether the deleted application servers are normal according to a preset configuration strategy; when the application server is normal, the application server list will be updated and the log file scan of the corresponding application server will be restarted.

6. The transaction log collection device according to claim 5, characterized in that, The sorting module includes: The target log file unit is used to determine the target log file based on the log output time and the transaction time of each log file. The output message queue unit is used to put the sorted target log files into the output message queue.

7. The transaction log collection device according to claim 5, characterized in that, Also includes: The tail transaction time determination module is used to determine the tail transaction time based on the transaction time of the last log file placed into the cache queue for each message queue.

8. The transaction log collection device according to claim 5, characterized in that, Also includes: The header message transaction time determination module is used to determine the transaction time of the header log file in the cache queue as the header message transaction time; or The header message transaction time is determined based on the transaction time of the first log file placed into the cache queue for each message queue.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the transaction log collection method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the transaction log collection method according to any one of claims 1 to 4.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the transaction log collection method according to any one of claims 1 to 4.

12. A transaction log collection system, characterized in that, include: The transaction log collection device according to any one of claims 5-8; Each application server is used to generate log files based on the transaction time and transaction message, and send the log files to the transaction log collection device according to the log scanning interval; The transaction playback device is used to consume sorted log files obtained from the output message queue of the transaction log collection device.

Images