Log processing method, apparatus, device, and computer-readable storage medium

Abstract

This invention relates to the field of financial technology (Fintech) and discloses a log processing method, comprising: acquiring a first log identifier of stored log data and a first sorting number corresponding to the first log identifier; determining a second sorting number corresponding to second log data in the log to be processed based on the first log identifier, the first sorting number, and a second log identifier of the log to be processed; sorting the second log data based on the second sorting number and the second log identifier to obtain sorted log data; and updating the stored log data based on the sorted log data. This invention also discloses a log processing apparatus, device, and computer-readable storage medium. By establishing a contextual association between newly stored log data and previously stored log data, this invention enables contextual location of log information based on log identifiers during retrieval, thereby improving the accuracy of log information retrieval.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and more particularly to a log processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, more and more technologies (big data, distributed systems, blockchain, artificial intelligence, etc.) are being applied in the financial sector. The traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. Currently, for service providers such as banks, the business servers in their backend business systems generate logs during operation. To facilitate log collection, storage, and retrieval, service providers also offer log collection and retrieval systems based on open-source software such as Filebeat (log collection agent), Logstash (log processing), and Elasticsearch (log storage).

[0003] When using keywords to retrieve logs in a log collection and retrieval system, existing log storage methods can only locate log information associated with that keyword (e.g., log information containing that keyword), resulting in low accuracy in log retrieval.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a log processing method, apparatus, device, and computer-readable storage medium, aiming to solve the technical problem of low accuracy in log retrieval caused by the existing method which can only retrieve related log information based on keywords.

[0006] To achieve the above objectives, the present invention provides a log processing method, the log processing method comprising the following steps:

[0007] Obtain the first log identifier of the stored log data and the first sorting number corresponding to the first log identifier;

[0008] Based on the first log identifier, the first sorting number, and the second log identifier of the log to be processed, determine the second sorting number corresponding to the second log data in the log to be processed.

[0009] Based on the second sorting number and the second log identifier, the second log data is sorted to obtain sorted log data;

[0010] The stored log data is updated based on the sorted log data.

[0011] Further, the step of determining the second sorting number corresponding to the log data in the log to be processed based on the first log identifier, the first sorting number, and the second log identifier of the log to be processed includes:

[0012] Determine whether there is a common log identifier between the first log identifier and the second log identifier;

[0013] If there is a common log identifier between the first log identifier and the second log identifier, then based on the first sorting number, obtain the third sorting number corresponding to the common log identifier in the stored log data;

[0014] The second sorting number is determined based on the third sorting number.

[0015] Further, the step of determining the second sorting number based on the third sorting number includes:

[0016] If there are no different log identifiers between the first log identifier and the second log identifier, then the second sorting number is determined based on the third sorting number;

[0017] If there are different log identifiers between the first log identifier and the second log identifier, then the second sorting number is determined based on the third sorting number and the initial number.

[0018] Furthermore, since there are different log identifiers between the first log identifier and the second log identifier, the step of sorting the second log data in the log to be processed based on the second sorting number and the second log identifier includes:

[0019] Based on the same log identifier and the third sorting number, sort the log data corresponding to the same log identifier in the second log data;

[0020] Based on the different log identifiers and initial sequence numbers, the log data corresponding to different log identifiers in the second log data are sorted.

[0021] Further, the step of sorting the log data corresponding to the same log identifier in the second log data based on the same log identifier and the third sorting number includes:

[0022] If the same log identifier includes one, and the log data corresponding to the same log identifier includes multiple, then obtain the first log time corresponding to each of the same log identifiers, and sort the log data corresponding to the same log identifiers based on the first log time and the third sorting number;

[0023] If there are multiple identical log identifiers, and the log data corresponding to the identical log identifiers contains multiple identical log data with the same log identifier, then the second log time corresponding to each identical log data is obtained. The identical log data is sorted based on the sorting number corresponding to the identical log data in the third sorting number and the second log time. The remaining log data is sorted based on the sorting number corresponding to the remaining log data in the third sorting number, wherein the remaining log data is other log data besides the identical log data in the log data corresponding to the identical log identifier.

[0024] Furthermore, after determining whether there is a common log identifier between the first log identifier and the second log identifier, the method further includes:

[0025] If there is no identical log identifier between the first log identifier and the second log identifier, then the second sorting sequence number is determined based on the second log identifier and the initial sequence number.

[0026] Furthermore, before the step of obtaining the first log identifier of the stored log data and the first sorting number corresponding to the first log identifier, the log processing method further includes:

[0027] Based on the log path, obtain the business logs in the distributed business system;

[0028] The business logs are formatted and filtered to obtain filtered business logs.

[0029] The filtered business logs are parsed to obtain the logs to be processed.

[0030] Furthermore, the log processing method also includes:

[0031] Upon receiving a log request, obtain the log information, thread information, and terminal identifier corresponding to the log request, and generate a log identifier based on the thread information and terminal identifier;

[0032] Business logs are generated based on the log information and the log identifier, and the log identifier and the business logs are stored together.

[0033] Furthermore, the log processing method also includes:

[0034] Upon receiving a retrieval request, the stored log data is retrieved based on the keywords corresponding to the retrieval request to obtain the third log data corresponding to the keywords;

[0035] Obtain the third log identifier corresponding to the third log data, and obtain the target log data corresponding to the third log identifier and the sorting number corresponding to the target log data from the stored log data.

[0036] Furthermore, to achieve the above objectives, the present invention also provides a log processing apparatus, the log processing apparatus comprising:

[0037] The acquisition module is used to acquire the first log identifier of the stored log data and the first sorting number corresponding to the first log identifier;

[0038] The determining module is used to determine the second sorting number corresponding to the log data in the log to be processed based on the first log identifier, the first sorting number, and the second log identifier of the log to be processed.

[0039] The sorting module is used to sort the second log data in the log to be processed based on the second sorting sequence number and the second log identifier, so as to obtain sorted log data.

[0040] The update module is used to update the stored log data based on the sorted log data.

[0041] In addition, to achieve the above objectives, the present invention also provides a log processing device, the log processing device comprising: a memory, a processor, and a log processing program stored in the memory and executable on the processor, wherein the log processing program, when executed by the processor, implements the steps of the aforementioned log processing method.

[0042] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a log processing program, which, when executed by a processor, implements the steps of the aforementioned log processing method.

[0043] This invention obtains a first log identifier of the stored log data and a first sorting number corresponding to the first log identifier. Then, based on the first log identifier, the first sorting number, and a second log identifier of the log to be processed, it determines a second sorting number corresponding to the second log data in the log to be processed. Subsequently, based on the second sorting number and the second log identifier, it sorts the second log data to obtain sorted log data. Then, it updates the stored log data based on the sorted log data. This invention enables the storage of log data in the log to be processed according to the log identifier and sorting number of the stored log data, thereby establishing a contextual association between the newly stored log data and the previously stored log data. This allows for contextual location of log information based on the log identifier during retrieval, improving the accuracy of log information retrieval. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of the log processing device in the hardware operating environment involved in the embodiments of the present invention;

[0045] Figure 2 This is a flowchart illustrating the first embodiment of the log processing method of the present invention;

[0046] Figure 3 This is a schematic diagram of the functional modules in one embodiment of the log processing device of the present invention.

[0047] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0049] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a log processing device in the hardware operating environment involved in the embodiments of the present invention.

[0050] In this embodiment of the invention, the log processing device can be a PC, a central server of a distributed business system, etc. Figure 1 As shown, the log processing device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0051] Optionally, the log processing device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Among these, sensors may include light sensors, motion sensors, and other sensors.

[0052] Those skilled in the art will understand that Figure 1The log processing device structure shown does not constitute a limitation on the log processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0053] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a log processing program.

[0054] exist Figure 1 In the log processing device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the log processing program stored in the memory 1005.

[0055] In this embodiment, the log processing device includes: a memory 1005, a processor 1001, and a log processing program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the log processing program stored in the memory 1005, it performs the operations in the following log processing method.

[0056] The present invention also provides a log processing method, referring to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the log processing method of the present invention.

[0057] In this embodiment, the log processing method includes:

[0058] Step S100: Obtain the first log identifier of the stored log data and the first sorting number corresponding to the first log identifier;

[0059] When log storage is required, the terminal nodes in the distributed business system send log requests to the central server. The central server generates business logs and corresponding log identifiers based on the log requests, and stores the business logs in the storage area corresponding to the log path, so that the log processor in the central server can collect them. The generated business logs can be formatted as: [Application Name][LogId][Date and Time][Thread Name][Log Level]-[Message Text], where LogId is the log identifier.

[0060] Then, the log processor retrieves business logs based on this log path. It's worth noting that information can be sent to the log processor when generating business logs. The log processor uses this information to determine if there are currently unprocessed business logs stored, and then retrieves them based on this log path. Alternatively, the log processor can determine in real-time whether there are unprocessed business logs based on the log path; if so, it retrieves them based on this path. For example, the log path could be: ` / home / app / logs / * / error / *.log`, where `*` indicates matching all folders in the current directory, and `*.log` indicates matching all log files in the current folder that end with the `.log` suffix. The log processor then determines the logs to be processed based on the retrieved business logs.

[0061] In this embodiment, when a log to be processed is obtained, the first log identifier of the stored log data and the first sorting number corresponding to the first log identifier are obtained. The stored log data, the first log identifier, and the first sorting number are associated and stored in a preset storage area or storage medium.

[0062] It is understandable that if multiple log data entries correspond to the same log identifier in the first log identifier, the maximum sorting number corresponding to these multiple log data entries is used as the sorting number corresponding to the same log identifier. Similarly, if a log identifier in the first log identifier corresponds to only one log data entry, the sorting number of that log data entry is the sorting number corresponding to that log identifier that corresponds to only one log data entry, thus determining the first sorting number. For example, if the first log identifier includes strings A, B, and C, and the sorting numbers corresponding to string A include 1, 2, and 3, the sorting numbers corresponding to string B include 1, 2, 3, and 4, and the sorting number corresponding to string C includes 1, then the first sorting number includes the maximum sorting number 3 corresponding to string A, the maximum sorting number 4 corresponding to string B, and the sorting number 1 corresponding to string C.

[0063] Step S200: Based on the first log identifier, the first sorting number, and the second log identifier of the log to be processed, determine the second sorting number corresponding to the second log data in the log to be processed;

[0064] In this embodiment, when the first log identifier and the first sorting number are obtained, the second sorting number corresponding to the second log data is determined based on the first log identifier, the first sorting number, and the second log identifier of the second log data (all log data in the log to be processed). Specifically, for log identifiers that are the same between the first log identifier and the second log identifier, the second sorting number corresponding to the same log identifier is determined according to the sorting number in the first sorting number corresponding to the same log identifier. For log identifiers that are different between the first log identifier and the second log identifier, the second sorting number corresponding to them is determined according to the initial number. For example, if the first log identifier includes strings A, B, and C, the first sorting number includes the maximum sorting number 3 corresponding to string A, the maximum sorting number 4 corresponding to string B, and the sorting number 1 corresponding to string C; if the second log identifier includes strings A and E, then the second sorting number includes the sorting number 4 corresponding to string A (the maximum sorting number 3+1 corresponding to string A) and the initial number (e.g., 1).

[0065] Step S300: Sort the second log data based on the second sorting number and the second log identifier to obtain sorted log data;

[0066] In this embodiment, upon obtaining the second sorting number, the second log data is sorted according to the second sorting number and the second log identifier to obtain sorted log data. Specifically, if there is only one second log identifier, the second log data is directly sorted according to the second sorting number. If there are multiple second log identifiers, the log data corresponding to each second log identifier is sorted according to the second sorting number. For example, if the second log identifiers include strings A and E, and the second sorting number includes the sorting number 4 corresponding to string A (the maximum sorting number 3+1 corresponding to string A) and the initial number (e.g., 1), then the log data corresponding to string A in the second log data is sorted according to the sorting number 4, and the log data corresponding to string E in the second log data is sorted according to the initial number.

[0067] Step S400: Update the stored log data based on the sorted log data.

[0068] In this embodiment, after obtaining the sorted log data, the stored log data is updated based on the sorted log data, that is, the sorted log data, the second log identifier, and the sorting sequence number corresponding to the sorted log data are associated and stored in the preset storage area or storage medium corresponding to the stored log data.

[0069] It should be noted that if there is no stored log data, the first log identifier and the first sorting number will both be empty or 0. Therefore, the second sorting number corresponding to the second log data is determined directly based on the second log identifier.

[0070] Since there is a certain correlation between log data generated by the same thread on the same terminal node, a contextual association can be established between newly stored log data and previously stored log data by using the sorting sequence number and log identifier. During retrieval, multiple log data corresponding to a specific log identifier can be retrieved based on the sorting sequence number and log identifier, thereby achieving contextual location of logs and obtaining contextual log data associated with the retrieved log data, thus improving the accuracy of log information retrieval.

[0071] The log processing method proposed in this embodiment obtains a first log identifier of the stored log data and a first sorting number corresponding to the first log identifier. Then, based on the first log identifier, the first sorting number, and a second log identifier of the log to be processed, a second sorting number corresponding to the second log data in the log to be processed is determined. Subsequently, based on the second sorting number and the second log identifier, the second log data is sorted to obtain sorted log data. Then, the stored log data is updated based on the sorted log data. This realizes the storage of log data in the log to be processed according to the log identifier and sorting number of the stored log data, thereby establishing a contextual association between the newly stored log data and the previously stored log data. It can realize contextual positioning of log information based on the log identifier during retrieval, thus improving the accuracy of log information retrieval.

[0072] Based on the first embodiment, a second embodiment of the log processing method of the present invention is proposed. In this embodiment, step S200 includes:

[0073] Step S210: Determine whether there is a common log identifier between the first log identifier and the second log identifier;

[0074] Step S220: If there is a common log identifier between the first log identifier and the second log identifier, then based on the first sorting number, obtain the third sorting number corresponding to the common log identifier in the stored log data;

[0075] Step S230: Determine the second sorting number based on the third sorting number.

[0076] Further, in one embodiment, step S230 includes:

[0077] Step S231: If there are no different log identifiers between the first log identifier and the second log identifier, then determine the second sorting number based on the third sorting number;

[0078] Step S232: If there are different log identifiers between the first log identifier and the second log identifier, then the second sorting number is determined based on the third sorting number and the initial number.

[0079] In this embodiment, when the first log identifier and the first sorting number are obtained, it is determined whether there is a common log identifier between the first log identifier and the second log identifier. If there is a common log identifier between the first log identifier and the second log identifier, then based on the first sorting number, the third sorting number corresponding to the common log identifier in the stored log data is obtained. For example, the first log identifier includes strings A, B, and C, and the first sorting number includes the maximum sorting number 3 corresponding to string A, the maximum sorting number 4 corresponding to string B, and the sorting number 1 corresponding to string C; the second log identifier includes string A, then the third sorting number includes the sorting number 4 corresponding to string A (the maximum sorting number 3+1 corresponding to string A).

[0080] Then, the second sorting number is determined based on the third sorting number. Specifically, if there are no different log identifiers between the first log identifier and the second log identifier, the second sorting number is determined based on the third sorting number; for example, if the second log identifier only includes the string A, then the third sorting number is the second sorting number, that is, the second sorting number is the sorting number 4 corresponding to the string A. If there are different log identifiers between the first log identifier and the second log identifier, the second sorting number is determined based on the third sorting number and the initial number; for example, if the second log identifier includes the strings A and E, then the second sorting number includes the third sorting number (the sorting number 4 corresponding to the string A) and the initial number.

[0081] It should be noted that if there are multiple identical log identifiers, the second sorting number corresponding to each identical log identifier is determined based on the third sorting number corresponding to the identical log identifier. If there are multiple different log identifiers, the sorting number corresponding to each different log identifier in the second sorting number is the initial number.

[0082] Furthermore, in one embodiment, after step S210, the method further includes: if there is no identical log identifier between the first log identifier and the second log identifier, then the second sorting sequence number is determined based on the second log identifier and the initial sequence number.

[0083] In this embodiment, if there is no identical log identifier between the first log identifier and the second log identifier, the second sorting number is determined based on the second log identifier and the initial sequence number. Specifically, the initial sequence number is used as the second sorting number.

[0084] The log processing method proposed in this embodiment determines whether there is a common log identifier between the first log identifier and the second log identifier. If there is a common log identifier between the first log identifier and the second log identifier, a third sorting number corresponding to the common log identifier in the stored log data is obtained based on the first sorting number. Then, based on the third sorting number, the second sorting number is determined. This method can accurately determine the second sorting number based on the first log identifier, the first sorting number, and the second log identifier, thereby improving the sorting accuracy of the second log data and further improving the accuracy of log information retrieval.

[0085] Based on the second embodiment, a third embodiment of the log processing method of the present invention is proposed. In this embodiment, there are different log identifiers between the first log identifier and the second log identifier. Step S300 includes:

[0086] Step S310: Sort the log data corresponding to the same log identifier in the second log data based on the same log identifier and the third sorting number;

[0087] Step S320: Sort the log data corresponding to different log identifiers in the second log data based on the different log identifiers and the initial sequence number.

[0088] In this embodiment, if there are different log identifiers between the first log identifier and the second log identifier, the log data corresponding to the same log identifier and the log data corresponding to different log identifiers are split. Based on the same log identifier and the third sorting number, the log data corresponding to the same log identifier in the second log data is sorted. At the same time, based on different log identifiers and the initial number, the log data corresponding to different log identifiers in the second log data is sorted to achieve accurate sorting of the second log data.

[0089] Further, in one embodiment, step S310 includes:

[0090] Step S311: If the same log identifier includes one and the log data corresponding to the same log identifier includes multiple, then obtain the first log time corresponding to each of the same log identifiers, and sort the log data corresponding to the same log identifiers based on the first log time and the third sorting number.

[0091] Step S312: If there are multiple identical log identifiers, and multiple identical log data with the same log identifier exist in the log data corresponding to the identical log identifiers, then obtain the second log time corresponding to each identical log data, sort the identical log data based on the sorting number corresponding to the identical log data in the third sorting number and the second log time, and sort the remaining log data based on the sorting number corresponding to the remaining log data in the third sorting number, wherein the remaining log data are other log data besides the identical log data in the log data corresponding to the identical log identifiers.

[0092] In this embodiment, since the log request sent by the terminal node may correspond to multiple log messages, the obtained business logs may include multiple entries, resulting in multiple log data entries in the logs to be processed. Consequently, there may be multiple identical log identifiers between the first log identifier and the second log identifier. When sorting the log data corresponding to the same log identifier, it is first determined whether there are multiple identical log identifiers.

[0093] If there is only one identical log identifier, then it is determined whether there are multiple log data corresponding to the same log identifier. If there are multiple log data corresponding to the same log identifier, the first log time corresponding to that identifier is obtained, which is the date and time of each log data corresponding to the same identifier. The log data corresponding to the same identifier is then sorted based on the first log time and the third sorting number. For example, if the log data corresponding to the same log identifier includes log data a and log data b, the first log time order is log data b, then log data a, and the third sorting number is 4, then the sorting number of log data b is 4, and the sorting number of log data a is 5. Conversely, if there is only one log data corresponding to the same log identifier, then the log data is sorted based on the third sorting number. For example, if the log data corresponding to the same log identifier includes log data a, and the third sorting number is 4, then the sorting number of log data a is 4.

[0094] If there are multiple identical log identifiers, then determine whether there are multiple identical log data entries with the same identifier in the log data corresponding to the same log identifier.

[0095] If there are multiple log data with the same log identifier in the log data corresponding to the same log identifier, then the second log time corresponding to each log data with the same identifier is obtained. The log data with the same identifier is sorted based on the sorting number corresponding to the log data with the same identifier in the third sorting number and the second log time. That is, the log data with the same identifier is sorted according to the order of the second log time, and then sorted according to the sorting number corresponding to the log data with the same identifier in the third sorting number. The remaining log data is sorted based on the sorting number corresponding to the remaining log data in the third sorting number. The remaining log data is the other log data in the log data corresponding to the same log identifier besides the log data with the same identifier. For example, the second log identifier includes strings A, B, and E, and the same log identifier is strings A and B. The third sorting number includes sorting number 4 corresponding to string A and sorting number 8 corresponding to string B. The log identifier corresponding to log data c and log data d is string A, and the log identifier corresponding to log data e is string B. That is, log data c and log data d are log data with the same identifier. If the chronological order of the second log times is log data c and log data d, then the sorting number of log data c is 4, the sorting number of log data d is 5, and the sorting number of log data c is 9.

[0096] If there are no multiple log data entries with the same log identifier among the log data corresponding to the same log identifier, then the log data corresponding to the same log identifier are directly sorted based on the third sorting number. For example, the second log identifier includes strings A, B, and E, the same log identifiers are strings A and B, the third sorting number includes sorting number 4 corresponding to string A and sorting number 8 corresponding to string B, the log identifier corresponding to log data f is string A, and the log identifier corresponding to log data g is string B, then the sorting number of log data f is 4, and the sorting number of log data g is 8.

[0097] It is understandable that when sorting log data corresponding to different log identifiers, it is necessary to determine whether there are multiple different log identifiers.

[0098] If there is only one different log identifier, then determine whether there are multiple log data corresponding to each different log identifier. If there are multiple log data corresponding to different log identifiers, then obtain the third log time corresponding to each different log identifier, i.e., the time and date in each log data corresponding to each different log identifier, and sort the log data corresponding to each different log identifier based on the third log time and the initial sequence number. That is, sort the log data corresponding to each different log identifier according to the chronological order of the third log time and the initial sequence number. If there is only one log data corresponding to each different log identifier, then use the initial sequence number as the sorting sequence number for the log data corresponding to each different log identifier.

[0099] If there are multiple different log identifiers, then determine whether there are multiple log data entries with the same identifier in the log data corresponding to the different log identifiers.

[0100] If multiple log data entries with the same log identifier exist across different log identifiers, then the fourth log time corresponding to each log data entry with the same identifier is obtained. The log data entries with the same identifier are then sorted based on the initial sequence number and the fourth log time. Specifically, they are sorted according to the chronological order of the fourth log time, followed by the initial sequence number. The remaining log data, which consists of all log data except those with the same identifier across different log identifiers, is then sorted directly based on the initial sequence number. If no multiple log data entries with the same log identifier exist across different log identifiers, then the log data across different log identifiers is sorted directly based on the initial sequence number.

[0101] It should be noted that if there are no identical log identifiers between the first log identifier and the second log identifier, the sorting method of the second log data is similar to the sorting method of log data corresponding to different log identifiers, and will not be elaborated here.

[0102] The log processing method proposed in this embodiment sorts the log data corresponding to the same log identifier in the second log data based on the same log identifier and the third sorting number, and sorts the log data corresponding to different log identifiers in the second log data based on different log identifiers and the initial number. This method can accurately sort the second log data according to the third sorting number and the initial number in the second sorting number, further improving the accuracy of log information retrieval.

[0103] Based on the first embodiment, a fourth embodiment of the log processing method of the present invention is proposed. In this embodiment, before step S100, the log processing method further includes:

[0104] Step S500: Based on the log path, obtain the business logs from the distributed business system;

[0105] Step S600: Perform formatted filtering on the business logs to obtain filtered business logs;

[0106] Step S700: Parse the filtered business logs to obtain the logs to be processed.

[0107] In this embodiment, the business log corresponding to the log path is obtained, and the business log is formatted and filtered to obtain the filtered business log. For example, the filtered business log is: "[OrderSystem][2ec99bc404eef445985560e998ed6d0b][2019-11-18 15:23:40][thread-179][ERROR]-[zhangsan create order, orderId:100011558]".

[0108] Then, the filtered business logs are parsed to obtain the logs to be processed. That is, the business logs are parsed according to a preset format to obtain standardized log data that meets the format requirements. For example, the log data in the logs to be processed is as follows:

[0109]

[0110] It is understandable that after obtaining the business logs, the business logs corresponding to the log path are deleted to prevent duplicate business logs from being obtained subsequently.

[0111] The log processing method proposed in this embodiment obtains business logs from a distributed business system based on log paths; then, it performs formatted filtering on the business logs to obtain filtered business logs; subsequently, it parses the filtered business logs to obtain the logs to be processed. By processing the business logs, logs to be processed that meet the format requirements are obtained, improving the parsing efficiency of business logs and thus improving the sorting efficiency of log data. Simultaneously, by acquiring business logs in real time, the timeliness and accuracy of business logs are improved.

[0112] Based on the first embodiment, a fifth embodiment of the log processing method of the present invention is proposed. In this embodiment, the log processing method further includes:

[0113] Step S800: Upon receiving a log request, obtain the log information, thread information, and terminal identifier corresponding to the log request, and generate a log identifier based on the thread information and terminal identifier;

[0114] Step S900: Generate a business log based on the log information and the log identifier, and store the log identifier and the business log together.

[0115] In this embodiment, when a log request is received, the log information, thread information and terminal identifier corresponding to the log request are obtained, and a log identifier is generated based on the thread information and terminal identifier, so that the same terminal identifier and thread information correspond to the same log identifier.

[0116] Then, business logs are generated based on log information and log identifiers, so that the log identifiers of business logs corresponding to the same terminal identifier and thread information are the same, and the log identifiers and the business logs are associated and stored, that is, the log identifiers and the business logs are associated and stored according to the log path.

[0117] The log processing method proposed in this embodiment obtains the log information, thread information, and terminal identifier corresponding to the log request when a log request is received, and generates a log identifier based on the thread information and terminal identifier. Then, it generates business logs based on the log information and log identifiers, and stores the log identifiers and business logs in association. By generating log identifiers based on thread information and terminal identifiers, business logs with the same terminal identifier and thread information have the same log identifier. Furthermore, when sorting business logs, they can be sorted according to the log identifiers, thus reflecting the contextual association between log data corresponding to the same terminal and the same thread. During retrieval, the contextual location of log information can be achieved based on the log identifiers, improving the accuracy of log information retrieval.

[0118] Based on the above embodiments, a sixth embodiment of the log processing method of the present invention is proposed. In this embodiment, the log processing method further includes:

[0119] Step a: Upon receiving a retrieval request, the stored log data is retrieved based on the keywords corresponding to the retrieval request to obtain the third log data corresponding to the keywords;

[0120] Step b: Obtain the third log identifier corresponding to the third log data, and obtain the target log data corresponding to the third log identifier and the sorting number corresponding to the target log data from the stored log data.

[0121] In this embodiment, when a search request is received, the keyword corresponding to the search request is determined, and the stored log data is searched according to the keyword to obtain the third log data corresponding to the keyword. That is, the third log data is the log data in the stored log data that includes the keyword.

[0122] Then, the third log identifier corresponding to the third log data is obtained, and the target log data corresponding to the third log identifier and the sorting number corresponding to the target log data are obtained from the stored log data. The target log data and the sorting number corresponding to the target log data are output as the search result corresponding to the search request, thereby realizing the context search of logs.

[0123] The log processing method proposed in this embodiment retrieves stored log data based on keywords corresponding to the retrieval request when a retrieval request is received, thereby obtaining third log data corresponding to the keywords; then, it obtains a third log identifier corresponding to the third log data, and retrieves the target log data corresponding to the third log identifier and the sorting number corresponding to the target log data from the stored log data, thereby realizing contextual retrieval of log information and improving the accuracy of log information retrieval.

[0124] This invention also provides a log processing device, as described in the embodiments of the present invention. Figure 3 The log processing device includes:

[0125] The acquisition module 100 is used to acquire the first log identifier of the stored log data and the first sorting number corresponding to the first log identifier;

[0126] The determining module 200 is used to determine the second sorting number corresponding to the log data in the log to be processed based on the first log identifier, the first sorting number, and the second log identifier of the log to be processed.

[0127] The sorting module 300 is used to sort the second log data in the log to be processed based on the second sorting sequence number and the second log identifier, so as to obtain sorted log data.

[0128] The update module 400 is used to update the stored log data based on the sorted log data.

[0129] Optionally, the determining module 200 is also used for:

[0130] Determine whether there is a common log identifier between the first log identifier and the second log identifier;

[0131] If there is a common log identifier between the first log identifier and the second log identifier, then based on the first sorting number, obtain the third sorting number corresponding to the common log identifier in the stored log data;

[0132] The second sorting number is determined based on the third sorting number.

[0133] Optionally, the determining module 200 is also used for:

[0134] If there are no different log identifiers between the first log identifier and the second log identifier, then the second sorting number is determined based on the third sorting number;

[0135] If there are different log identifiers between the first log identifier and the second log identifier, then the second sorting number is determined based on the third sorting number and the initial number.

[0136] Optionally, if there are different log identifiers between the first log identifier and the second log identifier, the sorting module 300 is further configured to:

[0137] Based on the same log identifier and the third sorting number, sort the log data corresponding to the same log identifier in the second log data;

[0138] Based on the different log identifiers and initial sequence numbers, the log data corresponding to different log identifiers in the second log data are sorted.

[0139] Optionally, the sorting module 300 is also used for:

[0140] If the same log identifier includes one, and the log data corresponding to the same log identifier includes multiple, then obtain the first log time corresponding to each of the same log identifiers, and sort the log data corresponding to the same log identifiers based on the first log time and the third sorting number;

[0141] If there are multiple identical log identifiers, and the log data corresponding to the identical log identifiers contains multiple identical log data with the same log identifier, then the second log time corresponding to each identical log data is obtained. The identical log data is sorted based on the sorting number corresponding to the identical log data in the third sorting number and the second log time. The remaining log data is sorted based on the sorting number corresponding to the remaining log data in the third sorting number, wherein the remaining log data is other log data besides the identical log data in the log data corresponding to the identical log identifier.

[0142] Optionally, the determining module 200 is also used for:

[0143] If there is no identical log identifier between the first log identifier and the second log identifier, then the second sorting sequence number is determined based on the second log identifier and the initial sequence number.

[0144] Optionally, the log processing apparatus further includes:

[0145] The log acquisition module is used to acquire business logs from the distributed business system based on the log path.

[0146] The filtering module is used to perform formatted filtering processing on the business logs to obtain filtered business logs.

[0147] The parsing module is used to parse and process the filtered business logs to obtain the logs to be processed.

[0148] Optionally, the log processing apparatus further includes:

[0149] The information acquisition module is used to acquire the log information, thread information and terminal identifier corresponding to the log request when a log request is received, and to generate a log identifier based on the thread information and terminal identifier.

[0150] The generation module is used to generate business logs based on the log information and the log identifier, and to store the log identifier and the business logs in association.

[0151] Optionally, the log processing apparatus further includes:

[0152] The retrieval module is used to retrieve stored log data based on the keywords corresponding to the retrieval request when a retrieval request is received, so as to obtain the third log data corresponding to the keywords;

[0153] The identifier acquisition module is used to acquire the third log identifier corresponding to the third log data, and to acquire the target log data corresponding to the third log identifier and the sorting number corresponding to the target log data from the stored log data.

[0154] The methods executed by the above-mentioned program modules can be referred to in the various embodiments of the log processing method of the present invention, and will not be repeated here.

[0155] Furthermore, embodiments of the present invention also propose a computer-readable storage medium.

[0156] The computer-readable storage medium stores a log processing program, which, when executed by a processor, implements the steps of the log processing method described above.

[0157] The method implemented when the log processing program running on the processor is executed can be referred to in various embodiments of the log processing method of the present invention, and will not be repeated here.

[0158] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0159] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0161] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A log processing method characterized by, The log processing method comprises the following steps: obtaining a first log identifier of stored log data and a first sorting sequence number corresponding to the first log identifier; determining a second sorting sequence number corresponding to second log data in the to-be-processed log based on the first log identifier, the first sorting sequence number, and a second log identifier of the to-be-processed log; sorting the second log data based on the second sorting sequence number and the second log identifier to obtain sorted log data; updating the stored log data based on the sorted log data; wherein the step of determining the second sorting sequence number corresponding to the log data in the to-be-processed log based on the first log identifier, the first sorting sequence number, and the second log identifier of the to-be-processed log comprises: determining whether there is a same log identifier between the first log identifier and the second log identifier; if there is a same log identifier between the first log identifier and the second log identifier, obtaining a third sorting sequence number corresponding to the same log identifier in the stored log data based on the first sorting sequence number; determining the second sorting sequence number based on the third sorting sequence number; if there is no same log identifier between the first log identifier and the second log identifier, determining the second sorting sequence number based on the second log identifier and an initial sequence number; wherein the step of determining the second sorting sequence number based on the third sorting sequence number comprises: if there is no different log identifier between the first log identifier and the second log identifier, determining the second sorting sequence number based on the third sorting sequence number; if there is a different log identifier between the first log identifier and the second log identifier, determining the second sorting sequence number based on the third sorting sequence number and an initial sequence number.

2. The log processing method of claim 1, wherein, if there is a different log identifier between the first log identifier and the second log identifier, the step of sorting the second log data based on the second sorting sequence number and the second log identifier comprises: sorting log data corresponding to the same log identifier in the second log data based on the same log identifier and the third sorting sequence number; sorting log data corresponding to the different log identifier in the second log data based on the different log identifier and an initial sequence number.

3. The log processing method of claim 2, wherein, the step of sorting log data corresponding to the same log identifier in the second log data based on the same log identifier and the third sorting sequence number comprises: if the same log identifier includes one and the log data corresponding to the same log identifier includes multiple, obtaining a first log time corresponding to each same log identifier, and sorting the log data corresponding to the same log identifier based on the first log time and the third sorting sequence number; if the same log identifier includes one and the log data corresponding to the same log identifier includes multiple, obtaining a first log time corresponding to each same log identifier, and sorting the log data corresponding to the same log identifier based on the first log time and the third sorting sequence number; If the same log identifier includes multiple, and the same log identifier corresponding log data exists in the same log identifier multiple same identifier log data, the second log time corresponding to each same identifier log data is obtained, the same identifier log data is sorted based on the third sorting sequence number corresponding to the same identifier log data and the second log time, and the remaining log data is sorted based on the sorting sequence number corresponding to the remaining log data in the third sorting sequence number, wherein the remaining log data is other log data in the same log identifier corresponding log data except the same identifier log data.

4. The log processing method of claim 1, wherein, Before the step of obtaining the first log identifier of the stored log data and the first sorting sequence number corresponding to the first log identifier, the log processing method further comprises: Based on the log path, the business log in the distributed business system is obtained; The business log is subjected to a format filtering processing operation to obtain a filtered business log; The filtered business log is subjected to a parsing processing to obtain the to-be-processed log.

5. The log processing method of claim 1, wherein, The log processing method further comprises: When receiving a log request, the log information, thread information and terminal identifier corresponding to the log request are obtained, and the log identifier is generated based on the thread information and terminal identifier; The business log is generated based on the log information and the log identifier, and the log identifier and the business log are associatedly stored.

6. The log processing method according to any one of claims 1 to 5, characterized in that, The log processing method further comprises: When receiving a search request, the stored log data is searched based on the keyword corresponding to the search request to obtain third log data corresponding to the keyword; The third log data corresponding to the third log identifier is obtained, and the target log data corresponding to the third log identifier and the sorting sequence number corresponding to the target log data are obtained in the stored log data.

7. A log processing apparatus characterized by comprising: The log processing device comprises: An obtaining module is configured to obtain a first log identifier of stored log data and a first sorting sequence number corresponding to the first log identifier; A determining module is configured to determine a second sorting sequence number corresponding to log data in a to-be-processed log based on the first log identifier, the first sorting sequence number and a second log identifier of the to-be-processed log; A sorting module is configured to sort second log data in the to-be-processed log based on the second sorting sequence number and the second log identifier to obtain sorted log data; An updating module is configured to update the stored log data based on the sorted log data; The determining module is further configured to: determine whether there is a same log identifier between the first log identifier and the second log identifier; if there is a same log identifier between the first log identifier and the second log identifier, acquire a third sorting sequence number corresponding to the same log identifier in the stored log data based on the first sorting sequence number; determine the second sorting sequence number based on the third sorting sequence number; and if there is no same log identifier between the first log identifier and the second log identifier, determine the second sorting sequence number based on the second log identifier and an initial sequence number. The determining module is further configured to: if there is no different log identifier between the first log identifier and the second log identifier, determine the second sorting sequence number based on the third sorting sequence number; and if there is a different log identifier between the first log identifier and the second log identifier, determine the second sorting sequence number based on the third sorting sequence number and an initial sequence number.

8. A log processing device characterized by comprising: The log processing device comprises a memory, a processor, and a log processing program stored in the memory and executable on the processor, and the log processing program, when executed by the processor, implements the steps of the log processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a log processing program, and the log processing program, when executed by the processor, implements the steps of the log processing method according to any one of claims 1 to 6.

Images