Log processing method, device and electronic equipment
By automatically calling the standby server to assist the log server in processing logs, the log server is solved, and the log server is inefficient when processing a large number of logs is achieved, achieving more efficient log processing and timely participation of the standby server.
Patent Information
- Application Number
- CN202211482416.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-11-24
AI Technical Summary
In the prior art, when processing a large number of system logs, the log server is unable to process all logs in time, resulting in inefficient processing and lack of effective solutions.
By detecting the number of logs received by the log server, if the number is greater than the target threshold, the backup server in the server cluster will be automatically called to assist the log server in processing the logs and realize load sharing.
It improves log processing efficiency, reduces manual participation, enhances the timeliness of call of standby servers, and solves the problem of low log processing efficiency.
Smart Images

Figure CN115866067B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of log processing, and in particular to a log processing method, device and electronic device. Background Art
[0002] In the prior art, when the log server processes the system log, if the number of logs to be processed is large, the log server may not be able to process all the logs to be processed in time. On this basis, usually only when the operation and maintenance personnel actively find that the load of the log server is high, will they manually call other servers to assist the log server in processing the logs to be processed.
[0003] However, since it is impossible for operation and maintenance personnel to detect the load status of the log server at any time, the timeliness of the intervention of the backup server cannot be effectively guaranteed, resulting in the problem of low system log processing efficiency.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a log processing method, device and electronic device to at least solve the technical problem of low log processing efficiency in the prior art.
[0006] According to one aspect of an embodiment of the present application, a log processing method is provided, including: obtaining, through a log server, multiple logs generated by a target system within a preset time period; detecting whether the number of the multiple logs is greater than a target threshold, wherein the target threshold represents the maximum number of logs that the log server can process under the current load; when the number of the multiple logs is greater than the target threshold, calling at least one standby server from a server cluster, and controlling the at least one standby server and the log server to process the multiple logs.
[0007] Furthermore, the log processing method also includes: performing format unification processing on each log to obtain multiple first logs, wherein the format of the message header of each first log is the target message header format, the format of the message body of each first log is the target message body format, the message header at least includes the function description information corresponding to the first log, and the message body at least includes the business logic information corresponding to the first log; identifying sensitive information in each first log and encrypting the sensitive information to obtain multiple second logs, wherein the sensitive information at least includes user privacy information; identifying redundant logs in the multiple second logs, and removing redundant logs from the multiple second logs to obtain at least one target log, wherein the redundant logs are abnormal logs or duplicate logs in the multiple second logs.
[0008] Furthermore, the log processing method also includes: detecting whether there is an abnormal log with a missing target identification field in multiple second logs, wherein the target identification field is used to represent the unique identification of each second log; if there is an abnormal log in the multiple second logs, determining that the abnormal log is a redundant log.
[0009] Furthermore, the log processing method also includes: detecting whether each second log is a log to be paired, wherein the log to be paired is a log that needs to be paired with an associated log, and there is a preset association relationship between the associated log and the log to be paired; if the second log is a log to be paired, detecting whether the second log is paired successfully; if the second log fails to be paired, determining that the second log is a redundant log.
[0010] Furthermore, the log processing method also includes: detecting whether there are at least two third logs in the multiple second logs, wherein the at least two third logs have the same target log information, and the target log information is the information in the log information of the third log except the log generation time; when there are at least two third logs in the multiple second logs, sorting the at least two third logs according to the log generation time of each third log to obtain a sorting result; determining at least one fourth log from the at least two third logs according to the sorting result, and determining that the fourth log is a redundant log.
[0011] Furthermore, the log processing method also includes: before detecting whether the number of multiple logs is greater than a target threshold, obtaining multiple historical logs processed by the log server within a target historical time period, wherein the target historical time period is composed of multiple historical sub-time periods, each historical sub-time period corresponds to at least one historical log among the multiple historical logs, and the length of each historical sub-time period is the same as the length of a preset time period; counting the number of historical logs corresponding to each historical sub-time period; and determining the threshold corresponding to each historical sub-time period based on the number of historical logs corresponding to each historical sub-time period.
[0012] Furthermore, the log processing method also includes: determining a target historical sub-time period corresponding to a preset time period from multiple historical sub-time periods, wherein the target historical sub-time period and the preset time period are two time periods with the same start time and end time under different time periods; determining a threshold corresponding to the target historical sub-time period as a target threshold, and detecting whether the number of multiple logs is greater than the target threshold.
[0013] Furthermore, the log processing method also includes: after obtaining multiple logs generated by the target system within a preset time period through the log server, obtaining the memory usage of the log server; detecting whether the memory usage is greater than a preset threshold; when the memory usage is greater than the preset threshold, calling at least one backup server from the server cluster to process multiple logs with the log server.
[0014] According to another aspect of an embodiment of the present application, a log processing device is also provided, including: an acquisition module, used to acquire multiple logs generated by a target system within a preset time period through a log server; a detection module, used to detect whether the number of multiple logs is greater than a target threshold, wherein the target threshold represents the maximum number of logs that the log server can process under the current load; a calling module, used to call at least one standby server from a server cluster when the number of multiple logs is greater than the target threshold, and control the at least one standby server and the log server to process the multiple logs.
[0015] According to another aspect of an embodiment of the present application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned log processing method.
[0016] In the present application, when the number of logs detected is greater than the target threshold, a backup server is automatically called to assist the log server in processing logs. The log server obtains multiple logs generated by the target system within a preset time period, and then detects whether the number of multiple logs is greater than the target threshold. When the number of multiple logs is greater than the target threshold, at least one backup server is called from the server cluster, and at least one backup server is controlled to process multiple logs with the log server. The target threshold represents the maximum number of logs that the log server can process under the current load.
[0017] From the above content, it can be seen that the present application detects the number of logs received by the log server, and when the number of logs is greater than the target threshold, automatically calls the backup server to assist the log server in processing the logs. Since the entire process does not require human participation, it not only reduces labor costs, but also improves the timeliness of calling the backup server, thereby achieving the technical effect of improving log processing efficiency.
[0018] It can be seen that the technical solution of the present application achieves the purpose of automatically calling the backup server to assist the log server in processing logs when the number of logs received by the log server is too large, thereby achieving the technical effect of improving the timeliness of the backup server's participation in log processing, and further solving the technical problem of low log processing efficiency in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 is a flowchart of an optional log processing method according to an embodiment of the present application;
[0021] Figure 2 is a flow chart of a method for determining a threshold value corresponding to each historical sub-time period based on historical logs;
[0022] Figure 3 is a flow chart of an optional method for determining a target threshold according to an embodiment of the present application;
[0023] Figure 4 is a flowchart of an optional log processing process according to an embodiment of the present application;
[0024] Figure 5 is a schematic diagram of an optional log processing device according to an embodiment of the present application;
[0025] Figure 6 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] In addition, it should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.
[0029] Example 1
[0030] According to an embodiment of the present application, an embodiment of a log processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0031] Figure 1 is a flow chart of an optional log processing method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:
[0032] Step S101: obtaining multiple logs generated by a target system within a preset time period through a log server.
[0033] In step S101, the log server is a server in the server cluster for processing logs. In addition, the target system includes but is not limited to the business system, the channel system and the platform system, wherein the business system includes but is not limited to the system for carrying out the bookkeeping business, the system for carrying out the loan credit business, the system for carrying out the account opening business, and the system for carrying out the transfer business. The channel system includes but is not limited to the channel system for submitting regulatory information. The platform system includes but is not limited to the SMS platform system and the user management platform system.
[0034] In addition, the above-mentioned preset time period can be customized, for example, one hour, one day or one week.
[0035] In an optional embodiment, a log management system can serve as the execution entity of the log processing method in the embodiment of the present application, wherein the log management system includes at least the above-mentioned target system, log server and backup server, and both the backup server and the log server belong to the above-mentioned server cluster.
[0036] Step S102: Detect whether the number of the multiple logs is greater than a target threshold.
[0037] In step S102, the log management system can centrally collect multiple logs generated by the target system within a preset time period through systems such as Kafka, and store all collected logs in the log server, and then the log server will count the number of all received logs to obtain the number of logs.
[0038] Furthermore, after obtaining the number of logs, the log server will determine whether the number of logs is greater than a target threshold, wherein the target threshold represents the maximum number of logs that the log server can process under the current load. For example, the log server will determine the maximum number of logs that can be processed based on the current processor usage and memory usage, and determine the maximum number of logs as the target threshold.
[0039] Step S103: when the number of the multiple logs is greater than the target threshold, at least one standby server is called from the server cluster, and the at least one standby server and the log server are controlled to process the multiple logs.
[0040] In step S103, when the log server detects that the number of logs is greater than the target threshold, it means that the log server can no longer efficiently process all the logs under the current load. Therefore, the log management system will call at least one backup server from the server cluster and control at least one backup server to process the logs together with the log service server.
[0041] It should be noted that the log server can calculate the difference between the number of logs and the target threshold, and then determine the number of backup servers that need to be called based on the difference. For example, if the difference is in the first interval, one backup server needs to be called. If the difference is in the second interval, two backup servers need to be called, where the minimum value of the second interval is greater than the maximum value of the first interval.
[0042] Based on the contents of the above steps S101 to S103, it can be known that in the present application, when the number of logs detected is greater than the target threshold, the backup server is automatically called to assist the log server in processing the logs, and the log server obtains multiple logs generated by the target system within a preset time period, and then detects whether the number of multiple logs is greater than the target threshold. When the number of multiple logs is greater than the target threshold, at least one backup server is called from the server cluster, and at least one backup server is controlled to process multiple logs with the log server. Wherein, the target threshold represents the maximum number of logs that the log server can process under the current load.
[0043] From the above content, it can be seen that the present application detects the number of logs received by the log server, and when the number of logs is greater than the target threshold, automatically calls the backup server to assist the log server in processing the logs. Since the entire process does not require human participation, it not only reduces labor costs, but also improves the timeliness of calling the backup server, thereby achieving the technical effect of improving log processing efficiency.
[0044] It can be seen that the technical solution of the present application achieves the purpose of automatically calling the backup server to assist the log server in processing logs when the number of logs received by the log server is too large, thereby achieving the technical effect of improving the timeliness of the backup server's participation in log processing, and further solving the technical problem of low log processing efficiency in the prior art.
[0045] In an optional embodiment, Figure 2 A flow chart of a method for determining the threshold value corresponding to each historical sub-time period of the log server according to the historical log is shown. Figure 2 As shown, the following steps are included:
[0046] Step S201 , before detecting whether the number of the plurality of logs is greater than a target threshold, obtaining a plurality of historical logs processed by the log server within a target historical time period.
[0047] In step S201, the target historical time period is composed of multiple historical sub-time periods, each historical sub-time period corresponds to at least one historical log among multiple historical logs, and the duration of each historical sub-time period is the same as the duration of the preset time period.
[0048] For example, the target historical time period is the past natural week, and each historical sub-time period is one hour long. Therefore, a natural week includes a total of 24*7=168 historical sub-time periods, and each historical sub-time period corresponds to at least one historical log.
[0049] Step S202: Count the number of historical logs corresponding to each historical sub-time period.
[0050] Still taking the above example, the log management system calculates the number of historical logs corresponding to each historical sub-time period, and obtains the number of historical logs corresponding to each historical sub-time period. For example, the number of historical logs corresponding to 8:00-9:00 on Monday is 1,000, the number of historical logs corresponding to 13:00-14:00 on Monday is 500, the number of historical logs corresponding to 10:00-11:00 on Wednesday is 1,200, and the number of historical logs corresponding to 15:00-16:00 on Sunday is 100.
[0051] Step S203: determining a threshold corresponding to each historical sub-time period according to the number of historical logs corresponding to each historical sub-time period.
[0052] Optionally, after obtaining the number of historical logs corresponding to each historical sub-time period, the log management system can determine the threshold value corresponding to each historical sub-time period. For example, the log management system can directly determine that the threshold value corresponding to 8:00-9:00 on Monday is 1000, the threshold value corresponding to 13:00-14:00 on Monday is 500, the threshold value corresponding to 10:00-11:00 on Wednesday is 1200, and the threshold value corresponding to 15:00-16:00 on Sunday is 100.
[0053] In addition, in an optional embodiment, the log management system can also adjust the number of historical logs corresponding to each historical sub-time period to obtain the threshold value corresponding to each historical sub-time period. For example, the log management system can directly determine that the threshold value corresponding to 8:00-9:00 on Monday is 1000+100=1100, the threshold value corresponding to 13:00-14:00 on Monday is 500+50=550, the threshold value corresponding to 10:00-11:00 on Wednesday is 1200+120=1320, and the threshold value corresponding to 15:00-16:00 on Sunday is 100+10=110.
[0054] In an optional embodiment, Figure 3 FIG. 2 shows a flow chart of an optional method for determining a target threshold according to an embodiment of the present application. Figure 3 As shown, the following steps are included:
[0055] Step S301: determining a target historical sub-time period corresponding to a preset time period from a plurality of historical sub-time periods.
[0056] In step S301, the target historical sub-time period and the preset time period are two time periods with the same start time and end time in different time periods.
[0057] For example, since the length of the preset time period is the same as the length of each historical sub-time period, if the preset time period is 8:00-9:00 on Monday this week, the target historical sub-time period is 8:00-9:00 last Monday; if the preset time period is 10:00-11:00 on Wednesday this week, the target historical sub-time period is 10:00-11:00 last Wednesday.
[0058] It is easy to notice that this week and last week are different time periods, but the start and end times of the preset time period and the target historical sub-time period are the same. In addition, the time period can be a natural year, a natural month, a natural day, etc. in addition to a natural week, and this application does not make any special restrictions on this.
[0059] Step S302: determine the threshold corresponding to the target historical sub-time period as the target threshold, and detect whether the number of the multiple logs is greater than the target threshold.
[0060] Optionally, in step S302, assuming that the target historical sub-time period is 8:00-9:00 last Monday, the corresponding threshold value is 1000, which means that the log number processed by the log server at 8:00-9:00 last Monday is about 1000. Those skilled in the art know that there is time periodicity in the system during operation, when the log number generated by a system at 8:00-9:00 last Monday is about 1000, if no special circumstances occur, the log number generated by the system at 8:00-9:00 this Monday should also be about 1000. On this basis, according to the technical scheme of the present application, 1000 can be determined as the target threshold value corresponding to the preset time period, if the log management system finds that the log number currently received by the log server is greater than 1000, it means that some special circumstances have occurred, thereby causing the log number to increase, in order to ensure that the log processing efficiency is not affected, the log management system can automatically call the standby server from the server cluster to participate in the process of log processing at this time, thereby sharing the processing pressure for the log server.
[0061] It is easy to notice that through the above solution, as long as the log management system finds that the number of logs in the same time period increases, the log management system will call the backup server to participate in log processing, which not only shares the burden of the log server, but also for the front-end users, when they initiate business processing requests in any time period, the perceived business response speed is stable, thereby achieving the effect of improving user experience.
[0062] In an optional embodiment, after obtaining multiple logs generated by the target system within a preset time period through the log server, the log management system can also obtain the memory usage of the log server and detect whether the memory usage is greater than a preset threshold. When the memory usage is greater than the preset threshold, the log management system calls at least one backup server from the server cluster to process multiple logs with the log server.
[0063] Optionally, the preset threshold can be customized, for example, 80%, 75%, 90%. Assuming that the preset threshold is set to 80%, when the log management system detects that the memory usage of the log server exceeds 80%, the log management system will call at least one backup server from the server cluster to process multiple logs together with the log server, thereby achieving the effect of the backup server sharing the log processing pressure for the log server.
[0064] In an optional embodiment, the log management system also controls at least one backup server and the log server to process multiple logs. Figure 4 As shown, the log processing process includes at least the following steps:
[0065] In step S401, the log management system first unifies the format of each log through the backup server and the log server to obtain multiple first logs, wherein the format of the message header of each first log is the target message header format, the format of the message body of each first log is the target message body format, the message header at least contains the function description information corresponding to the first log, and the message body at least contains the business logic information corresponding to the first log.
[0066] In step S401, since the target system may include multiple systems, the backup server and the log server need to process the logs from different systems in a unified format, specifically, the message header and the message body. The message header is used to define the header information of the main implementation function of the log; the message body is used to define the main business and logic implemented by the log. As shown in Table 1, the formatted log is in the following form:
[0067] Table 1
[0068]
[0069]
[0070] It should be noted that standardizing the log format can improve log management efficiency.
[0071] Step S402: The log management system identifies sensitive information in each first log through the backup server and the log server, and encrypts the sensitive information to obtain multiple second logs, wherein the sensitive information at least includes user privacy information.
[0072] In step S402, sensitive information includes but is not limited to ID number, mobile phone number, bank card number, email address, unified social information code, user name, home address and other information.
[0073] By encrypting sensitive information, the security of user privacy information can be improved.
[0074] Step S403, the log management system identifies redundant logs in the plurality of second logs through the backup server and the log server, and removes the redundant logs from the plurality of second logs to obtain at least one target log, wherein the redundant logs are abnormal logs or duplicate logs in the plurality of second logs.
[0075] It should be noted that by identifying and removing redundant logs, the number of logs that the backup server and the log server need to process in the subsequent process can be reduced, thereby improving the efficiency of log processing.
[0076] In an optional embodiment, the log management system detects whether there is an abnormal log with a missing target identification field in the plurality of second logs through the log server and the backup server, and determines that the abnormal log is a redundant log when there is an abnormal log in the plurality of second logs. The target identification field is used to represent a unique identifier of each second log.
[0077] Optionally, the YwtransCode transaction code in Table 1 above is the target identification field, which is the unique identifier of the log and is used to concatenate logs, track business flows and data flows, for example, to track transaction links as mobile APP-ESC-knowledge graph-core-external software. If the target identification field is missing in a log, the log management system will determine that the log is a redundant log.
[0078] In an optional embodiment, the log management system further detects whether each second log is a log to be paired through the log server and the backup server, and if the second log is a log to be paired, detects whether the second log is paired successfully, and if the second log pairing fails, determines that the second log is a redundant log. The log to be paired is a log that needs to be paired with the associated log, and there is a preset association relationship between the associated log and the log to be paired.
[0079] Optionally, in actual applications, some logs have matching rules. For example, logs with the create-over field are logs to be matched. These logs need to be matched with associated logs with preset association relationships before they can be processed normally by the server. For example, in the common reconciliation scenario in the financial industry, the billing log needs to be matched with the sales document log before it can be determined whether the reconciliation can be successful. On this basis, if a log to be matched cannot be matched successfully (for example, the server cannot find its corresponding associated log), the log management system will determine that the log to be matched is a redundant log.
[0080] In an optional embodiment, the log management system further detects whether there are at least two third logs in the plurality of second logs through the log server and the backup server. In the case that there are at least two third logs in the plurality of second logs, the log management system sorts the at least two third logs according to the log generation time of each third log to obtain a sorting result, and determines at least one fourth log from the at least two third logs according to the sorting result, and determines that the fourth log is a redundant log. Wherein, at least two third logs have the same target log information, and the target log information is the information in the log information of the third log except the log generation time.
[0081] Optionally, each of the at least two third logs only has a different log generation time, and all other log information is the same. Therefore, the log management system can determine that the at least two third logs are logs with duplicate message content. In order to improve log processing efficiency, the log management system can only retain one third log, and determine the other third logs as fourth logs and delete them. Which third log to retain can be determined based on the log generation time, for example, retaining the third log with the earliest log generation time, or retaining the third log with the latest log generation time.
[0082] It should be noted that when the log server and the backup server process multiple logs, they first identify the redundant logs in the multiple logs and delete these redundant logs from the multiple logs. This can not only reduce the number of logs that need to be processed later, but also improve the overall log processing efficiency, thereby achieving the effect of improving business response speed and helping to improve user experience.
[0083] Example 2
[0084] According to an embodiment of the present application, a log processing device is also provided, wherein: Figure 5 is a schematic diagram of an optional log processing device according to an embodiment of the present application, such as Figure 5 As shown, the device includes: an acquisition module 501, which is used to acquire multiple logs generated by the target system within a preset time period through a log server; a detection module 502, which is used to detect whether the number of multiple logs is greater than a target threshold, wherein the target threshold represents the maximum number of logs that the log server can process under the current load; a calling module 503, which is used to call at least one backup server from the server cluster when the number of multiple logs is greater than the target threshold, and control the at least one backup server and the log server to process the multiple logs.
[0085] It should be noted that the above-mentioned acquisition module 501, detection module 502 and calling module 503 correspond to steps S101 to S103 in the above-mentioned embodiment 1, and the examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.
[0086] Optionally, the calling module further includes: a first processing unit, a first identification unit and a second identification unit. The first processing unit is used to perform format unification processing on each log to obtain multiple first logs, wherein the format of the message header of each first log is the target message header format, the format of the message body of each first log is the target message body format, the message header at least contains the function description information corresponding to the first log, and the message body at least contains the business logic information corresponding to the first log; the first identification unit is used to identify sensitive information in each first log and encrypt the sensitive information to obtain multiple second logs, wherein the sensitive information at least includes user privacy information; the second identification unit is used to identify redundant logs in multiple second logs and remove redundant logs from multiple second logs to obtain at least one target log, wherein redundant logs are abnormal logs or duplicate logs in multiple second logs.
[0087] Optionally, the second identification unit further includes: a first detection subunit and a first determination subunit. The first detection subunit is used to detect whether there is an abnormal log with a missing target identification field in the multiple second logs, wherein the target identification field is used to represent the unique identification of each second log; the first determination subunit is used to determine that the abnormal log is a redundant log when there is an abnormal log in the multiple second logs.
[0088] Optionally, the second identification unit further includes: a second detection subunit, a third detection subunit, and a second determination subunit. The second detection subunit is used to detect whether each second log is a log to be paired, wherein the log to be paired is a log that needs to be paired with an associated log, and there is a preset association relationship between the associated log and the log to be paired; the third detection subunit is used to detect whether the second log is successfully paired if the second log is a log to be paired; the second determination subunit is used to determine that the second log is a redundant log if the second log fails to be paired.
[0089] Optionally, the second identification unit further includes: a fourth detection subunit, a sorting subunit and a third determination subunit. The fourth detection subunit is used to detect whether there are at least two third logs in the plurality of second logs, wherein at least two third logs have the same target log information, and the target log information is the information in the log information of the third log except the log generation time; the sorting subunit is used to sort the at least two third logs according to the log generation time of each third log to obtain a sorting result when there are at least two third logs in the plurality of second logs; the third determination subunit is used to determine at least one fourth log from the at least two third logs according to the sorting result, and determine that the fourth log is a redundant log.
[0090] Optionally, the log processing device further includes: a first acquisition module, a statistics module and a determination module. The first acquisition module is used to acquire multiple historical logs processed by the log server within a target historical time period, wherein the target historical time period is composed of multiple historical sub-time periods, each historical sub-time period corresponds to at least one historical log among the multiple historical logs, and the duration of each historical sub-time period is the same as the duration of the preset time period; the statistics module is used to count the number of historical logs corresponding to each historical sub-time period; and the determination module is used to determine the threshold value corresponding to each historical sub-time period according to the number of historical logs corresponding to each historical sub-time period.
[0091] Optionally, the detection module further includes: a first determination unit and a second determination unit. The first determination unit is used to determine a target historical sub-time period corresponding to a preset time period from a plurality of historical sub-time periods, wherein the target historical sub-time period and the preset time period are two time periods with the same start time and end time under different time periods; the second determination unit is used to determine a threshold corresponding to the target historical sub-time period as a target threshold, and detect whether the number of multiple logs is greater than the target threshold.
[0092] Optionally, the log processing device further includes: a second acquisition module, a first detection module, and a first calling module. The second acquisition module is used to acquire the memory usage of the log server; the first detection module is used to detect whether the memory usage is greater than a preset threshold; the first calling module is used to call at least one backup server and the log server from the server cluster to process multiple logs when the memory usage is greater than the preset threshold.
[0093] Example 3
[0094] According to an embodiment of the present application, an electronic device embodiment is also provided, wherein: Figure 6 is a schematic diagram of an optional electronic device according to an embodiment of the present application, such as Figure 6 As shown, the electronic device includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented:
[0095] Acquire multiple logs generated by the target system within a preset time period through a log server; detect whether the number of multiple logs is greater than a target threshold, wherein the target threshold represents the maximum number of logs that the log server can process under the current load; when the number of multiple logs is greater than the target threshold, call at least one standby server from the server cluster, and control at least one standby server and the log server to process the multiple logs.
[0096] Optionally, when executing the program, the processor also implements the following steps: performing format unification processing on each log to obtain multiple first logs, wherein the format of the message header of each first log is the target message header format, and the format of the message body of each first log is the target message body format, the message header at least includes functional description information corresponding to the first log, and the message body at least includes business logic information corresponding to the first log; identifying sensitive information in each first log and encrypting the sensitive information to obtain multiple second logs, wherein the sensitive information at least includes user privacy information; identifying redundant logs in the multiple second logs, and removing redundant logs from the multiple second logs to obtain at least one target log, wherein the redundant logs are abnormal logs or duplicate logs in the multiple second logs.
[0097] Optionally, when the processor executes the program, the following steps are also implemented: detecting whether there is an abnormal log with a missing target identification field among multiple second logs, wherein the target identification field is used to represent the unique identification of each second log; if there is an abnormal log among multiple second logs, determining that the abnormal log is a redundant log.
[0098] Optionally, when the processor executes the program, the following steps are further implemented: detecting whether each second log is a log to be paired, wherein the log to be paired is a log that needs to be paired with an associated log, and there is a preset association relationship between the associated log and the log to be paired; if the second log is a log to be paired, detecting whether the second log is paired successfully; if the second log pairing fails, determining that the second log is a redundant log.
[0099] Optionally, when executing the program, the processor further implements the following steps: detecting whether there are at least two third logs among the multiple second logs, wherein the at least two third logs have the same target log information, and the target log information is information in the log information of the third log except the log generation time; when there are at least two third logs among the multiple second logs, sorting the at least two third logs according to the log generation time of each third log to obtain a sorting result; determining at least one fourth log from the at least two third logs according to the sorting result, and determining that the fourth log is a redundant log.
[0100] Optionally, the processor also implements the following steps when executing the program: before detecting whether the number of multiple logs is greater than a target threshold, obtaining multiple historical logs processed by the log server within a target historical time period, wherein the target historical time period is composed of multiple historical sub-time periods, each historical sub-time period corresponds to at least one historical log among the multiple historical logs, and the length of each historical sub-time period is the same as the length of a preset time period; counting the number of historical logs corresponding to each historical sub-time period; and determining the threshold corresponding to each historical sub-time period based on the number of historical logs corresponding to each historical sub-time period.
[0101] Optionally, when executing the program, the processor also implements the following steps: determining a target historical sub-time period corresponding to a preset time period from multiple historical sub-time periods, wherein the target historical sub-time period and the preset time period are two time periods with the same start time and end time under different time periods; determining a threshold corresponding to the target historical sub-time period as a target threshold, and detecting whether the number of multiple logs is greater than the target threshold.
[0102] Optionally, the processor also implements the following steps when executing the program: after obtaining multiple logs generated by the target system within a preset time period through the log server, obtaining the memory usage of the log server; detecting whether the memory usage is greater than a preset threshold; when the memory usage is greater than the preset threshold, calling at least one backup server from the server cluster to process multiple logs with the log server.
[0103] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0104] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0109] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A log processing method, characterized in that: include: Obtain multiple logs generated by the target system within a preset time period through the log server; Detecting whether the number of the plurality of logs is greater than a target threshold, wherein the target threshold represents a maximum number of logs that the log server can process under a current load; In the case where the number of the plurality of logs is greater than the target threshold, calling at least one standby server from the server cluster, and controlling at least one of the standby servers and the log server to process the plurality of logs, wherein the log server calculates the difference between the number of the logs and the target threshold, and then determines the number of the standby servers to be called according to the difference; Wherein, controlling at least one of the standby servers and the log server to process the plurality of logs comprises: performing format unification processing on each of the logs to obtain a plurality of first logs, wherein the format of the message header of each of the first logs is the target message header format, the format of the message body of each of the first logs is the target message body format, the message header at least includes the function description information corresponding to the first log, and the message body at least includes the business logic information corresponding to the first log; identifying sensitive information in each of the first logs, and encrypting the sensitive information to obtain a plurality of second logs, wherein the sensitive information at least includes user privacy information; identifying redundant logs in the plurality of the second logs, and removing the redundant logs from the plurality of the second logs to obtain at least one target log, wherein the redundant logs are abnormal logs or duplicate logs in the plurality of the second logs; Wherein, identifying a redundant log among the plurality of second logs includes: detecting whether each of the second logs is a log to be paired, wherein the log to be paired is a log that needs to be paired with an associated log, and there is a preset association relationship between the associated log and the log to be paired; if the second log is the log to be paired, detecting whether the second log is successfully paired; if the second log fails to be paired, determining that the second log is the redundant log; Before detecting whether the number of multiple logs is greater than a target threshold, the method also includes: obtaining multiple historical logs processed by the log server within a target historical time period, wherein the target historical time period is composed of multiple historical sub-time periods, each of the historical sub-time periods corresponds to at least one of the multiple historical logs, and the length of each of the historical sub-time periods is the same as the length of the preset time period; counting the number of historical logs corresponding to each of the historical sub-time periods; and determining a threshold corresponding to each of the historical sub-time periods based on the number of historical logs corresponding to each of the historical sub-time periods.
2. The method according to claim 1, characterized in that Identifying redundant logs among the plurality of second logs, comprising: Detecting whether there is an abnormal log with a missing target identification field among the plurality of second logs, wherein the target identification field is used to represent a unique identifier of each second log; In the case that the abnormal log exists in a plurality of the second logs, the abnormal log is determined to be the redundant log.
3. The method according to claim 1, characterized in that Identifying redundant logs among the plurality of second logs, comprising: Detecting whether there are at least two third logs among the plurality of second logs, wherein at least two of the third logs have the same target log information, and the target log information is information in the log information of the third logs except the log generation time; In the case that there are at least two of the third logs in the plurality of the second logs, sorting the at least two third logs according to the log generation time of each of the third logs to obtain a sorting result; At least one fourth log is determined from at least two of the third logs according to the sorting result, and the fourth log is determined to be the redundant log.
4. The method according to claim 1, characterized in that Detecting whether the number of the plurality of logs is greater than a target threshold includes: Determine a target historical sub-time period corresponding to the preset time period from the plurality of historical sub-time periods, wherein the target historical sub-time period and the preset time period are two time periods with the same start time and end time in different time periods; A threshold corresponding to the target historical sub-time period is determined as the target threshold, and it is detected whether the number of the plurality of logs is greater than the target threshold.
5. The method according to claim 1, characterized in that After acquiring multiple logs generated by the target system within a preset time period through the log server, the method further includes: Obtaining the memory usage of the log server; Detecting whether the memory usage is greater than a preset threshold; When the memory usage rate is greater than the preset threshold, at least one of the standby servers and the log server is called from the server cluster to process the plurality of logs.
6. A log processing device, characterized in that: include: An acquisition module is used to acquire multiple logs generated by the target system within a preset time period through a log server; A detection module, used to detect whether the number of the plurality of logs is greater than a target threshold, wherein the target threshold represents the maximum number of logs that the log server can process under the current load; A calling module, used for calling at least one standby server from the server cluster when the number of the plurality of logs is greater than the target threshold, and controlling at least one of the standby servers and the log server to process the plurality of logs, wherein the log server calculates the difference between the number of the logs and the target threshold, and then determines the number of the standby servers to be called according to the difference; The calling module also includes: a first processing unit, which performs format unification processing on each of the logs to obtain multiple first logs, wherein the format of the message header of each of the first logs is the target message header format, the format of the message body of each of the first logs is the target message body format, the message header at least includes the function description information corresponding to the first log, and the message body at least includes the business logic information corresponding to the first log; a first identification unit, which identifies sensitive information in each of the first logs and encrypts the sensitive information to obtain multiple second logs, wherein the sensitive information at least includes user privacy information; a second identification unit, which identifies redundant logs in the multiple second logs and removes the redundant logs from the multiple second logs to obtain at least one target log, wherein the redundant logs are abnormal logs or duplicate logs in the multiple second logs; The second identification unit includes: a second detection subunit, used to detect whether each of the second logs is a log to be paired, wherein the log to be paired is a log that needs to be paired with an associated log, and there is a preset association relationship between the associated log and the log to be paired; a third detection subunit, used to detect whether the second log is successfully paired when the second log is the log to be paired; a second determination subunit, used to determine that the second log is the redundant log when the second log fails to be paired; The log processing device also includes: a first acquisition module, used to acquire multiple historical logs processed by the log server within a target historical time period, wherein the target historical time period is composed of multiple historical sub-time periods, each of the historical sub-time periods corresponds to at least one of the multiple historical logs, and the length of each of the historical sub-time periods is the same as the length of the preset time period; a statistical module, used to count the number of historical logs corresponding to each of the historical sub-time periods; and a determination module, used to determine a threshold value corresponding to each of the historical sub-time periods according to the number of historical logs corresponding to each of the historical sub-time periods.
7. An electronic device, characterized in that: The system comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the log processing method described in any one of claims 1 to 5.
Citation Information
Patent Citations
Log duplication eliminating processing method and device
CN106844143A
Log processing method and system
CN107544888A
Log monitoring method and device
CN112328463A
