Log processing method and device and electronic equipment

Abstract

The invention belongs to the technical field of computers, and provides a log processing method and device and electronic equipment. The method comprises the following steps: determining attribute information of a collected log; based on the attribute information, storing the log content of the log to a corresponding partition in the message middleware; and performing duplicate removal processing on the log data in each partition, and storing the log data subjected to duplicate removal into a database. According to the method, the redundant logs can be comprehensively deleted, the log deduplication accuracy is improved, and the occupation of the redundant logs on the database can be reduced.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, specifically to data processing technology, and in particular to a log processing method, device and electronic device. Background Art

[0002] Logs can record the status information of applications during operation, the operation behaviors of users during the use of applications, etc. By analyzing logs, valuable information can be mined. In order to avoid missing logs due to network problems, logs are usually collected in batches periodically. This method often leads to a large number of logs being collected repeatedly, resulting in redundant logs occupying storage space. Therefore, deleting redundant logs has become an urgent problem to be solved.

[0003] In the related art, redundant logs in each storage space are usually deleted. However, since the collected redundant logs may be distributed in different storage spaces, the related art cannot completely delete the redundant logs, which is not conducive to log analysis. Summary of the invention

[0004] The present application provides a log processing method, device and electronic device to solve the technical problem that redundant logs cannot be completely deleted.

[0005] A first aspect of an embodiment of the present application provides a log processing method, the method comprising: determining attribute information of a collected log; based on the attribute information, storing the log content of the log to a corresponding partition in a message middleware; deduplicating the log data in each partition, and storing the deduplicated log data in a database.

[0006] According to an embodiment of the present application, before determining the attribute information of the collected logs, the method further includes: collecting the logs based on pre-configured whitelist information and blacklist information.

[0007] According to an embodiment of the present application, based on the attribute information, the log content of the log is stored in a corresponding partition in the message middleware, including: determining the partition corresponding to the attribute information from the message middleware; based on the partition corresponding to the attribute information, storing a key-value pair consisting of the attribute information and the log content.

[0008] According to an embodiment of the present application, each partition includes a master copy and a slave copy, and the method further includes: if both the master copy and the slave copy receive the key-value pair, returning a reply code to the source terminal of the log, and the reply code is used to prompt that the key-value pair is stored successfully.

[0009] According to an embodiment of the present application, the deduplication processing of the log data in each partition includes: grouping the log data in each partition based on a sliding window to obtain multiple log groups; performing duplicate detection on the log data in the multiple log groups, and deleting the detected duplicate log data.

[0010] According to an embodiment of the present application, after deduplication processing is performed on the log data in each partition, the method further includes: based on preset rules, identifying log data with abnormal information from the deduplication log data; and moving the log data with abnormal information to a designated partition of the message middleware.

[0011] According to an embodiment of the present application, storing the deduplicated log data in a database includes: based on the configured table information, creating a data table corresponding to the configured table information in the database; and writing the deduplicated log data into the data table.

[0012] According to an embodiment of the present application, the method also includes: if the total amount of logs in the message middleware is the same as the number of collected logs, performing deduplication processing on the log data in each partition, and storing the deduplicated log data in a database; if the total amount of logs in the message middleware is different from the number of collected logs, generating a prompt message, and the prompt message is used to indicate that an abnormal situation occurs in the collected logs.

[0013] A second aspect of an embodiment of the present application provides a log processing device, which includes: a determination unit, used to determine attribute information of a collected log; a storage unit, used to store the log content of the log to a corresponding partition in a message middleware based on the attribute information; a processing unit, used to deduplicate the log data in each partition, and store the deduplicated log data in a database.

[0014] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory storing computer-readable instructions; and a processor executing the computer-readable instructions stored in the memory to implement the log processing method.

[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in an electronic device to implement the log processing method.

[0016] In multiple embodiments of the present application, based on the attribute information, the log content in the collected log is stored in the corresponding partition in the message middleware. Since the attribute information corresponding to the logs with the same log content is usually the same, it is possible to ensure that the logs with the same log content are stored in the same partition, so that when the log data in each partition is deduplicated, redundant logs can be fully deleted, and the accuracy of log deduplication can be improved. In addition, since the log data in each partition is deduplicated separately, the deduplication efficiency can be improved. By deduplicating the log data in each partition, the occupancy of the database by redundant logs can be reduced, and the analysis of the log can also be facilitated. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural diagram of an electronic device for implementing a log processing method provided in an embodiment of the present application.

[0018] Figure 2 It is a flow chart of the log processing method provided in the embodiment of the present application.

[0019] Figure 3 This is a flow chart of a log processing method provided by another embodiment of the present application.

[0020] Figure 4 It is a schematic diagram of the configuration interface of the log collection provided in the embodiment of the present application.

[0021] Figure 5 This is a flow chart of a log processing method provided by another embodiment of the present application.

[0022] Figure 6 This is a flow chart of a log processing method provided by another embodiment of the present application.

[0023] Figure 7 It is a functional module diagram of the log processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0027] In modern information systems, logs can record the status information of applications during operation, the operation behaviors of users during the use of applications, etc. By analyzing logs, valuable information can be mined. For example, by analyzing logs, it can assist operation and maintenance personnel to quickly troubleshoot; for another example, by analyzing logs, it can assist sales personnel to quickly understand user profile information.

[0028] In order to avoid missing log collection due to network problems, logs are usually collected in batches periodically. This method often results in a large number of logs being collected repeatedly, resulting in redundant logs occupying storage space and affecting log processing efficiency.

[0029] In the related art, redundant logs in each storage space are usually deleted. However, since the collected redundant logs may be distributed in different storage spaces, the related art cannot completely delete the redundant logs, which is not conducive to log analysis.

[0030] Based on this, an embodiment of the present application provides a log processing method, which stores logs with the same log content in the same partition, so that when deduplicating the log data in each partition, redundant logs can be completely deleted, thereby improving the accuracy of log deduplication.

[0031] In addition, related technologies make it difficult to accurately control the timing and scope of log collection, resulting in key log information not being collected in a timely or accurate manner, thereby affecting the effectiveness of log analysis.

[0032] Based on this, an embodiment of the present application provides another log processing method, which collects logs based on pre-configured whitelist information and blacklist information. Since the whitelist information can record information that needs to be collected, and the blacklist information can record information that does not need to be collected, the logs can be collected accurately, avoiding missing key logs or collecting irrelevant logs, which is facilitating log analysis.

[0033] like Figure 1 , which is a schematic diagram of the structure of an electronic device for implementing a log processing method provided in an embodiment of the present application.

[0034] In an embodiment of the present application, the log processing method is applied to one or more electronic devices 1. The electronic device 1 is a device that can execute computer-readable instructions to automatically perform numerical calculations and / or information processing, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0035] The electronic device 1 can be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0036] The electronic device 1 may include a network device and / or a user device, wherein the network device includes but is not limited to a single network electronic device, an electronic device group consisting of multiple network electronic devices, or a cloud consisting of a large number of hosts or network electronic devices based on cloud computing.

[0037] The network where the electronic device 1 is located includes, but is not limited to: the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0038] In the embodiment of the present application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions stored in the memory 12 and executable on the processor 13, such as a log processing program.

[0039] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 1 may also include input and output devices, network access devices, buses, etc.

[0040] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or a processor or any conventional processor, etc. The processor 13 is the computing core and control center of the electronic device 1, and uses various interfaces and lines to connect various parts of the entire electronic device 1, and execute the operating system of the electronic device 1 and various installed applications, program codes, etc.

[0041] The memory 12 may be an external memory and / or an internal memory of the electronic device 1. Furthermore, the memory 12 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0042] Combination Figure 2 , Figure 3 , Figure 5 , Figure 6 The memory 12 in the electronic device 1 stores computer-readable instructions, and the processor 13 can execute the computer-readable instructions stored in the memory 12 to implement the following Figure 2 , Figure 3 , Figure 5 , Figure 6 The multiple processes shown are used to implement the log processing method.

[0043] like Figure 2 , which is a flow chart of the log processing method provided by the embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The log processing method provided by the embodiment of the present application includes the following steps.

[0044] S201, determining attribute information of the collected logs.

[0045] In at least one embodiment of the present application, the attribute information includes, but is not limited to: log recording time, log level, address information corresponding to the source terminal of the log, server name corresponding to the consumption server of the log, function name, and location information of the log stored in the source terminal. Among them, the log level may include debug level, information level, warning level, error level, etc. The source terminal may represent the terminal device that generates the log, and the consumption server may represent the server that consumes the log.

[0046] In at least one embodiment of the present application, the electronic device can collect the log recording time, log level, address information corresponding to the source terminal of the log, and server name corresponding to the consumption server of the log from the message header of the log. The electronic device can collect the function name from the method body of the log. The electronic device uses the collection location of the log as location information.

[0047] S202: based on the attribute information, store the log content of the log into a corresponding partition in the message middleware.

[0048] In at least one embodiment of the present application, the log content may include information in the method body of the log. Different attribute information corresponds to different partitions. For example, if the log level of log A is the debug level, the partition corresponding to log A in the message middleware may be partition0; if the log level of log B is the error level, the partition corresponding to log A in the message middleware may be partition1. For another example, if the location information of log C is IP1, the partition corresponding to log C in the message middleware may be partition0; if the location information of log D is IP2, the partition corresponding to log D in the message middleware may be partition1.

[0049] In at least one embodiment of the present application, the electronic device stores the log content of the log to the corresponding partition in the message middleware based on the attribute information, including: determining the partition corresponding to the attribute information from the message middleware, and storing the key-value pairs consisting of the attribute information and the log content based on the partition corresponding to the attribute information.

[0050] The message middleware stores the correspondence between the attribute information and the partition identifier, and the partition identifier is used to uniquely identify different partitions. For example, the partition identifier topic0 partition0, the partition identifier topic0 partition1 and the partition identifier topic1partition0 represent different partitions respectively.

[0051] For example, the attribute information includes the log level and location information. The log level included in the attribute information s1 is the debug level, the location information included in the attribute information s1 is IP1, the log level included in the attribute information s2 is the error level, the location information included in the attribute information s2 is IP2, the partition corresponding to the attribute information s1 is topic0 partition0, and the partition corresponding to the attribute information s2 is topic0 partition1. If the log level of log A is the debug level, the location information of log A is IP1, the log level of log B is the error level, and the location information of log B is IP2. The log level (debug level) and location information (IP1) of log A are used as the first key key, the log content of log A is used as the first value value, and the first key-value pair corresponding to log A (including the first key key and the first value value) is stored in topic0 partition0. The log level (error level) and location information (IP2) of log B are used as the second key key, the log content of log B is used as the second value value, and the second key-value pair of log B (including the second key key and the second value value) is stored in topic0 partition1.

[0052] In this embodiment, the corresponding partition is determined by the attribute information. Since the logs with the same log content correspond to the same attribute information, it can ensure that the logs with the same log content are stored in the same partition.

[0053] In another embodiment, the electronic device stores the log content of the log in the corresponding partition in the message middleware based on the attribute information, and further includes: converting the attribute information into a hash value, determining the partition corresponding to the hash value from the message middleware, and storing a key-value pair consisting of the hash value and the log content based on the partition corresponding to the hash value. Different attribute information corresponds to different hash values. The corresponding relationship between the hash value and the partition identifier is stored in the message middleware.

[0054] Continuing with the above example, the log level (debug level) and location information (IP1) of log A are converted into the first hash value, and the log level (error level) and location information (IP2) of log B are converted into the second hash value. The first hash value and the log content of log A form a first key-value pair, and the second hash value and the log content of log B form a second key-value pair. The first key-value pair is stored in topic0 partition0, and the second key-value pair is stored in topic0 partition1.

[0055] In at least one embodiment of the present application, the key-value pairs are stored in sequence according to the order of arrangement of the log record time.

[0056] In at least one embodiment of the present application, each partition includes a master copy and a slave copy, and the method further includes: if both the master copy and the slave copy receive the key-value pair, the electronic device returns a response code to the source terminal of the log, and the response code is used to prompt that the key-value pair is successfully stored. In this embodiment, the response code is returned to prompt that the key-value pair is successfully stored only when both the master copy and the slave copy receive the key-value pair, which can ensure the successful storage and backup of the log and ensure the effectiveness of the log storage.

[0057] S203, performing deduplication processing on the log data in each partition, and storing the deduplicated log data in a database.

[0058] In at least one embodiment of the present application, the database may be a database that supports idempotent writing or transactional storage, for example, the database may include a MySQL database, an Oracle database, etc. This embodiment can avoid duplicate writing of log data to the database due to network or system failures by storing the deduplicated log data in a database that supports idempotent writing, thereby improving the effectiveness of deduplication. This embodiment can combine a two-phase commit (2PC) protocol (Two-phase Commit Protocol) by storing the deduplicated log data in a database that supports transactional storage, thereby ensuring that the deduplicated log data is successfully stored in the database, thereby avoiding log storage failures due to network problems or system failures.

[0059] In at least one embodiment of the present application, deduplication processing is performed on the log data in each partition, including: based on a sliding window, grouping the log data in each partition to obtain multiple log groups, performing duplication detection on the log data in the multiple log groups, and deleting the detected duplicate log data.

[0060] Among them, the sliding window can be a time window, and the length of the sliding window can be set to a preset duration. The preset duration can be set and adjusted according to needs. For example, the starting time point of the first sliding window is 9:00, and the preset duration is 1 hour, then the first sliding window can be 9:00-10:00, and the second sliding window can be 10:00-11:00.

[0061] In order to avoid missing log collection due to network problems, logs are usually collected in batches periodically. For example, the period set for the first log collection can be 9:00-9:30, and the period set for the second log collection can be 9:10-9:40. Therefore, if there is no problem with the network, the logs from 9:10-9:30 will be collected repeatedly. In order to delete redundant logs from 9:10-9:30, the preset duration can also be determined based on the period set when collecting logs. According to the above example, the preset duration can be set to the duration from 9:10 to 9:30, that is, 20 minutes. By determining the preset duration based on the period set when collecting logs, it can be ensured that logs with the same content are in the same log group.

[0062] Each partition corresponds to multiple log groups. The number of logs in each log group can be the same or different. For example, for the partition topic0 partition1, there can be 2 log groups. The first log group can include 100 logs, and the second log group can include 120 logs.

[0063] This embodiment groups the log data in each partition through a sliding window to obtain multiple log groups, and performs duplicate detection on the multiple log groups in parallel, which can improve the detection efficiency of redundant logs, thereby improving the deletion efficiency of redundant logs. In addition, since logs with the same content are stored in the same partition, and logs are stored according to the log recording time, logs with the same content are usually in the same log group, and then by performing duplicate detection on the log group, since the number of logs in the log group is relatively small, the detection of redundant logs can be further improved, thereby improving the deletion efficiency of redundant logs.

[0064] In at least one embodiment of the present application, storing deduplicated log data in a database includes: based on the configured table information, creating a data table corresponding to the configured table information in the database, and writing the deduplicated log data into the data table.

[0065] The configured table information may include but is not limited to: project name, database name, table name, and a separator between any two log data. The project name may indicate the project to which the log belongs, the database name may indicate the database where the log is stored, and the table name may indicate the name of the created data table.

[0066] This embodiment can create a data table corresponding to the configured table information in the database through the configured table information, so that when the database does not include the data table, the storage of deduplicated log data is realized.

[0067] In one embodiment, logs are collected from the local directory of the source terminal, the collected logs are stored in the server of the corresponding area, the logs in the servers corresponding to multiple areas are stored in the central server, and based on the attribute information of the logs, the logs are stored in the corresponding partitions in the message middleware, the log data in each partition is deduplicated, and the deduplicated log data is stored in the database. For example, the area where the source terminal Z1 and the source terminal Z2 are located is area Q1, and the area where the source terminal Z3, the source terminal Z4, and the source terminal Z5 are located is area Q2. The logs collected from the source terminal Z1 and the source terminal Z2 are stored in the server F1 corresponding to area Q1, and the logs collected from the source terminal Z3, the source terminal Z4, and the source terminal Z5 are stored in the server F2 corresponding to area Q2. Servers F1 and F2 store the logs in the central server respectively, and the central server can be a server corresponding to any specified area.

[0068] In a plurality of embodiments of the present application, based on the attribute information, the log content in the collected log is stored in the corresponding partition in the message middleware. Since the attribute information corresponding to the logs with the same log content is usually the same, it is possible to ensure that the logs with the same log content are stored in the same partition, so that when the log data in each partition is deduplicated, the redundant logs can be fully deleted, and the accuracy of log deduplication can be improved. In addition, since the log data in each partition is deduplicated separately, the deduplication efficiency can be improved. By deduplicating the log data in each partition, the occupancy of the database by redundant logs can be reduced, and the analysis of the logs can also be facilitated. For example, by log collection, abnormal conditions in the system can be captured and recorded in time, so that the operation and maintenance users can quickly locate system problems, reduce the impact of system downtime and service interruption, and can also help the operation and maintenance users to quickly analyze faults, thereby accelerating the repair process and improving the availability and stability of the service. For another example, by collecting logs related to system performance, bottlenecks in the system can be identified, such as high-load resources, slow queries, memory leaks, etc., thereby providing data support for system performance tuning. It can also monitor the health of each part of the system, detect potential performance problems in a timely manner, and perform preventive maintenance. For another example, the collected logs can be used to track user behavior, detect abnormal activities and potential security threats, and identify and respond to security incidents in a timely manner. For another example, during the development and testing phase, logs can help developers understand the code operation and verify the functional correctness.

[0069] like Figure 3 , is a flow chart of a log processing method provided by another embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The log processing method provided by the embodiment of the present application includes the following steps.

[0070] S301, collecting logs based on pre-configured whitelist information and blacklist information.

[0071] In at least one embodiment of the present application, the whitelist information and the blacklist information can be set according to the collection requirements. Figure 4 As shown, Figure 4 In the configuration interface of log collection provided in the embodiment of the present application, the user can enter whitelist information and blacklist information in the configuration interface of log collection. For example, the whitelist information can be as follows: Figure 4 The key shown is MY_PRODUCT_NAME_ENV, value is hago. When the user clicks the "Add" button at the whitelist of environment variables in the configuration interface of log collection, the input box of whitelist information can be displayed. When the user clicks the "Add" button at the blacklist of environment variables in the configuration interface of log collection, the input box of blacklist information can be displayed.

[0072] In at least one embodiment of the present application, the electronic device detects whether the log includes whitelist information. If the log includes whitelist information, the electronic device detects whether the log includes blacklist information. If the log does not include blacklist information, the log is collected. If the log does not include whitelist information, or the log includes blacklist information, the collection of the log is skipped. This embodiment can collect logs according to actual collection requirements through pre-configured whitelist information and blacklist information, thereby improving the personalization of log collection.

[0073] In at least one embodiment of the present application, a log agent is deployed on the source terminal of the log, logs are collected in real time, and the collected logs are sent to the message queue system. The log agent may include but is not limited to: Fluentd agent, Logstash agent, Filebeat agent.

[0074] S302: Determine the attribute information of the collected logs.

[0075] S303: based on the attribute information, store the log content of the log into a corresponding partition in the message middleware.

[0076] S304, performing deduplication processing on the log data in each partition, and storing the deduplicated log data in a database.

[0077] The details of steps S302-S304 can be found above. Figure 2 The detailed description of steps S201-S203 is not repeated here.

[0078] In this embodiment, through the pre-configured whitelist information and blacklist information, logs can be collected according to actual collection requirements, so that key logs can be collected, which is beneficial to log analysis. It can also avoid collecting irrelevant logs that affect log analysis.

[0079] like Figure 5 , is a flow chart of a log processing method provided by another embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The log processing method provided by the embodiment of the present application includes the following steps.

[0080] S501: Determine attribute information of the collected logs.

[0081] S502: based on the attribute information, store the log content of the log into a corresponding partition in the message middleware.

[0082] S503, performing deduplication processing on the log data in each partition, and storing the deduplicated log data in a database.

[0083] For details of steps S501-S503, please refer to the above Figure 2 The detailed description of steps S201-S203 is not repeated here.

[0084] S504: Based on preset rules, log data with abnormal information is identified from the deduplicated log data.

[0085] In at least one embodiment of the present application, the electronic device identifies log data with abnormal information from deduplicated log data based on preset rules, including: detecting whether the deduplicated log data includes setting information in the preset rules; if the deduplicated log data includes the setting information in the preset rules, determining the deduplicated log data as log data with abnormal information.

[0086] S505: Move the log data with exception information to a designated partition of the message middleware.

[0087] In at least one embodiment of the present application, log data with abnormal information is stored in a designated partition, and the log data stored in the designated partition is used to analyze abnormal behavior, system abnormality, and the like.

[0088] In this embodiment, through preset rules, log data with abnormal information can be accurately identified from the deduplicated log data, and then the log data with abnormal information can be moved to the designated partition of the message middleware. Since the designated partition specifically stores log data with abnormal information, it is beneficial for operation and maintenance users to analyze the logs.

[0089] like Figure 6, is a flow chart of a log processing method provided by another embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The log processing method provided by the embodiment of the present application includes the following steps.

[0090] S601: Determine attribute information of the collected logs.

[0091] S602: based on the attribute information, store the log content of the log into a corresponding partition in the message middleware.

[0092] For details of steps S601-S602, please refer to the above Figure 2 The detailed description of steps S201-S202 is not repeated here.

[0093] S603: Check whether the total amount of logs in the message middleware is the same as the amount of collected logs.

[0094] In at least one embodiment of the present application, if the total amount of logs in the message middleware is the same as the number of collected logs, execute S604; if the total amount of logs in the message middleware is different from the number of collected logs, execute S605.

[0095] S604, performing deduplication processing on the log data in each partition, and storing the deduplicated log data in a database.

[0096] For details of step S604, please refer to the above Figure 2 The detailed description of step S203 is not repeated here.

[0097] S605: Generate prompt information.

[0098] In at least one embodiment of the present application, the prompt information is used to indicate that an abnormal situation occurs in the collected log. If the total amount of logs in the message middleware is less than the number of collected logs, a prompt information indicating that the logs are lost is generated. If the total amount of logs in the message middleware is greater than the number of logs in the source terminal, a prompt information indicating that the logs are redundant is generated. Since logs are usually collected periodically (in order to deal with data loss problems caused by network problems), the number of collected logs is usually greater than the number of logs in the source terminal.

[0099] In this embodiment, when the total amount of logs in the message middleware is the same as the number of collected logs, it can be indicated that the collected logs do not have abnormal log loss, and then the log data in each partition is further deduplicated to avoid the inability to fully deduplicate the log data, thereby improving the effectiveness of deduplication. When the total amount of logs in the message middleware is different from the number of collected logs, a prompt message can be generated to indicate whether the logs are lost or not.

[0100] like Figure 7 , which is a functional module diagram of a log processing device provided in an embodiment of the present application. The log processing device 11 includes a determination unit 110, a storage unit 111, a processing unit 112, a collection unit 113, a sending unit 114, an identification unit 115, a mobile unit 116, and a generation unit 117. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be acquired by the processor 13 and can perform fixed functions, which are stored in the memory 12.

[0101] In one embodiment, the determination unit 110 is used to determine the attribute information of the collected log; the storage unit 111 is used to store the log content of the log to the corresponding partition in the message middleware based on the attribute information; the processing unit 112 is used to deduplicate the log data in each partition and store the deduplicated log data in the database.

[0102] In one embodiment, before determining the attribute information of the collected logs, the collecting unit 113 is used to collect the logs based on pre-configured whitelist information and blacklist information.

[0103] In one embodiment, the storage unit 111 is specifically used to: determine the partition corresponding to the attribute information from the message middleware; and store the key-value pair consisting of the attribute information and the log content based on the partition corresponding to the attribute information.

[0104] In one embodiment, each partition includes a master copy and a slave copy. The sending unit 114 is used to return a response code to the source terminal of the log if both the master copy and the slave copy receive the key-value pair. The response code is used to indicate that the key-value pair is stored successfully.

[0105] In one embodiment, the processing unit 112 is specifically used to: group the log data in each partition based on the sliding window to obtain multiple log groups; perform duplication detection on the log data in the multiple log groups, and delete the detected duplicate log data.

[0106] In one embodiment, after deduplication processing is performed on the log data in each partition, the identification unit 115 is used to identify log data with abnormal information from the deduplication log data based on preset rules; the moving unit 116 is used to move the log data with abnormal information to a specified partition of the message middleware.

[0107] In one embodiment, the processing unit 112 is specifically configured to: create a data table corresponding to the configured table information in the database based on the configured table information; and write the deduplicated log data into the data table.

[0108] In one embodiment, the processing unit 112 is also used to perform deduplication processing on the log data in each partition and store the deduplicated log data in the database if the total amount of logs in the message middleware is the same as the number of collected logs; the generation unit 117 is used to generate prompt information if the total amount of logs in the message middleware is different from the number of collected logs, and the prompt information is used to indicate that an abnormality has occurred in the collected logs.

[0109] In a plurality of embodiments of the present application, based on the attribute information, the log content in the collected log is stored in the corresponding partition in the message middleware. Since the attribute information corresponding to the logs with the same log content is usually the same, it is possible to ensure that the logs with the same log content are stored in the same partition, so that when the log data in each partition is deduplicated, redundant logs can be fully deleted, and the accuracy of log deduplication can be improved. In addition, since the log data in each partition is deduplicated separately, the deduplication efficiency can be improved. By deduplicating the log data in each partition, the occupancy of the database by redundant logs can be reduced, and the analysis of the log can also be facilitated.

[0110] If the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware to complete it through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the computer-readable instructions are executed by the processor, the steps of the above-mentioned method embodiments can be implemented.

[0111] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable files, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer-readable instruction codes, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM).

[0112] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer readable instructions can refer to Figure 2 , Figure 3 , Figure 5 , Figure 6 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0114] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0116] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present application is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0117] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A log processing method, characterized in that: The method comprises: Determine the attribute information of the collected logs; Based on the attribute information, the log content of the log is stored in a corresponding partition in the message middleware; De-duplicate the log data in each partition and store the de-duplicated log data in the database.

2. The log processing method according to claim 1, characterized in that: Before determining the attribute information of the collected log, the method further includes: Collect logs based on pre-configured whitelist and blacklist information.

3. The log processing method according to claim 1, characterized in that: Based on the attribute information, storing the log content of the log to a corresponding partition in the message middleware includes: Determine a partition corresponding to the attribute information from the message middleware; Based on the partition corresponding to the attribute information, a key-value pair consisting of the attribute information and the log content is stored.

4. The log processing method according to claim 3, characterized in that: Each partition includes a master copy and a slave copy, and the method further includes: If both the master copy and the slave copy receive the key-value pair, a response code is returned to the source terminal of the log, and the response code is used to prompt that the key-value pair is stored successfully.

5. The log processing method according to claim 1, characterized in that: The deduplication processing of the log data in each partition includes: Based on the sliding window, the log data in each partition is grouped to obtain multiple log groups; Duplicate detection is performed on the log data in the multiple log groups, and the detected duplicate log data is deleted.

6. The log processing method according to claim 1, characterized in that: After deduplication processing is performed on the log data in each partition, the method further includes: Based on preset rules, identifying log data with abnormal information from the deduplicated log data; The log data with the exception information is moved to a designated partition of the message middleware.

7. The log processing method according to claim 1, characterized in that: The storing of the deduplicated log data into the database includes: Based on the configured table information, creating a data table corresponding to the configured table information in the database; The deduplicated log data is written into the data table.

8. The log processing method according to any one of claims 1 to 7, characterized in that: The method further comprises: If the total amount of logs in the message middleware is the same as the number of the collected logs, perform deduplication processing on the log data in each partition, and store the deduplicated log data in the database; If the total amount of logs in the message middleware is different from the amount of the collected logs, a prompt message is generated, where the prompt message is used to indicate that an abnormality occurs in the collected logs.

9. A log processing device, characterized in that: The device comprises: A determination unit, used to determine attribute information of the collected logs; A storage unit, used to store the log content of the log to a corresponding partition in the message middleware based on the attribute information; The processing unit is used to perform deduplication processing on the log data in each partition and store the deduplication log data in the database.

10. An electronic device, characterized in that: include: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the log processing method according to any one of claims 1 to 8.

Images